Image processing device, image processing method, and program

ABSTRACT

There is provided an image processing device including a subject distance change determination unit configured to detect a temporal change of a distance from an imaging position to each subject present in an image and determine a tendency toward approach or recession of the each subject with respect to the imaging position on the basis of the detection, and a main subject determination unit configured to determine a main subject on the basis of the tendency toward approach or recession of the each subject determined by the subject distance change determination unit.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a Continuation Application of application Ser. No.14/655,621, filed Jun. 25, 2015, which is a national stage ofPCT/JP2013/080606, filed Nov. 12, 2013, which claims the benefit ofJapanese Patent Application JP 2013-001798 filed on Jan. 9, 2013, thecontents of which are incorporated herein by reference in theirentirety.

TECHNICAL FIELD

The present disclosure relates to an image processing device, an imageprocessing method, and a program for performing a process of determininga main subject in an image.

BACKGROUND ART

Recent digital still cameras and digital video cameras normally have afacial detection function, and have a function of optimally matchingvarious parameters (focus, brightness and the like) of the cameraaccording to facial position and area.

On the other hand, Patent Literature 1 discloses a technique in which auser designates and selects a “main subject” that is a target subject tobe tracked in a captured image.

In addition, if the techniques disclosed in Patent Literatures 2, 3, and4 described above are used, for example, subject tracking thatencompasses a frame of the whole body of an arbitrary subject can berealized.

In addition, there are also functions of controlling an optical systemand the like such as autofocus and automatic exposure such that adesired area in a captured image is detected and traced so as tooptimize the area.

As described above, technologies of tracking an image designated by auser as a main subject, for example, an image area such as a face in acaptured image, focusing on the face area, and the like are known.

CITATION LIST Patent Literature

Patent Literature 1: JP 2011-166305A

Patent Literature 2: JP 2011-146826A

Patent Literature 3: JP 2011-146827A

Patent Literature 4: JP 2011-160379A

SUMMARY OF INVENTION Technical Problem

However, in a captured image, a desired area that is subject to trackingor focusing, i.e., a “main subject,” is decided by a photographer bydirectly selecting one candidate from “a plurality of candidate areas”obtained from various detectors using any method at present.

In a state of holding a camera, for example, a main subject is chosenthrough an action of selecting an arbitrary face from a plurality offaces projected on a through image displayed on a screen (a monitoringimage of a subject displayed at times other than at a time ofmanipulating a shutter) on a touch panel. Alternatively, a subjectpresent in a predetermined area is set to be a main subject at a timedesignated by a user (half-pressing of a shutter or the like).

However, when a user interface is considered in actual use examples,there are cases in which the action of “selecting a main subject by aphotographer” itself is difficult.

For example, when a user wants to use a function of maintaining focus ona subject that is moving around, it is difficult for the user to selectthe subject using his or her finger holding a camera while targeting thesubject.

For example, there are cases in which designation is difficult due to aspeed of a reaction of a user to changes (movements) of a subject. Forexample, there is a case in which it is difficult to precisely designatea subject that is moving around on a screen of a through image.

In addition, in a situation in which a user holds a camera in his or herhand in the first place and turns the camera toward the subject tochoose the subject, it is difficult for the user to perform the actionof selecting a main subject using his or her finger on a screen.Particularly, in a situation in which a subject is tracked, for example,in which a user changes an imaging direction to cause a camera to movefor panning or tilting, it is almost not possible to select a mainsubject.

In addition, there is a case in which it is difficult for a use toselect a subject in resolution of a display screen in which a touchpanel is arranged.

In addition, there is also a case in which it is difficult to properlydesignate a desired subject depending on a size of the subject on adisplay screen in which a touch panel is arranged and a size (thickness)of a user's finger.

In addition, there is also a case in which it is difficult for a user toproperly designate a subject due to a time lag of a camera system, forexample, a time lag between actual scenery and a through image of acaptured image.

Furthermore, when such a designation manipulation is to be performedduring imaging and recording of a dynamic image, there may be caseswhere image shaking caused by an action of selecting a main subject maybe recorded as it is, or an user may be forced to take an action ofre-selection at the time of frame-out or tracking loss (failure) causedby temporary shielding, or the like.

As described above, hand-held type cameras have a problem in that theaction of selecting a main subject itself is difficult in many useexamples that require the function, which is stressful forphotographers.

Therefore, the present disclosure aims to realize a technology ofdetermining a target subject desired by a user such as a photographerand setting the subject as a main subject without an action of the userintentionally selecting the subject.

Solution to Problem

According to the present disclosure, an image processing device includesa subject distance change determination unit configured to detect atemporal change of a distance from an imaging position to each subjectpresent in an image and determine a tendency toward approach orrecession of the each subject with respect to the imaging position onthe basis of the detection, and a main subject determination unitconfigured to determine a main subject on the basis of the tendencytoward approach or recession of the each subject determined by thesubject distance change determination unit.

According to the present disclosure, an image processing method includesdetecting a temporal change of a distance from an imaging position toeach subject present in an image and determining a tendency towardapproach or recession of the each subject with respect to the imagingposition on the basis of the detection, and determining a main subjecton the basis of the determined tendency toward approach or recession ofthe each subject.

According to the present disclosure, a program causing an arithmeticprocessing device to perform the processes for achieving the imageprocessing device or the image processing method.

According to an embodiment of the present disclosure, the main subjectdetermination process of determining the main subject automatically inthe image data is performed.

For example, when a user holds an imaging apparatus to perform imaging,the user aims at an approaching subject or a receding subject in somecases. For example, there are cases in which a child, a friend, a pet,or the like is approaching a photographer holding a camera or cases inwhich a train, an airplane, a wild bird, or the like is receding from acamera. There are cases in which an approaching subject is set as animaging target or a receding subject is set as an imaging targetaccording to such use cases (imaging target). Thus, by determiningtendencies of approach or recession with respect to imaging positions ofsubjects and determining a main subject based on the tendency towardapproach or recession of each of the determined subjects, a subject thatthe user considers to be a principal or main figure can be estimated,and thus automatic main subject determination can be performedaccordingly.

Advantageous Effects of Invention

According to the present disclosure, a main subject is automaticallydetermined in a captured image, and thus it is not necessary for a usersuch as a photographer to perform an action of selecting the mainsubject. Accordingly, enhancement of product-added value includingimprovement of operability when imaging is performed using the imagingapparatus in which the image processing device of the present disclosureis mounted held in a hand, a reduction of stress on users, and furtherrealization of various functions caused by automatic main subjectdetermination can be realized.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram of a configuration example of an imageprocessing device of an embodiment of the present disclosure.

FIG. 2 is a flowchart of a main subject determination process of theimage processing device of the embodiment.

FIG. 3 is a block diagram of an imaging apparatus of a first embodiment.

FIG. 4 is an explanatory diagram of the overview of main subjectdetermination of the first embodiment.

FIG. 5 is an explanatory diagram of approach determination of the firstembodiment.

FIG. 6 is a flowchart of the approach determination and a main subjectdetermination process of the first embodiment.

FIG. 7 is a block diagram of an imaging apparatus of a secondembodiment.

FIG. 8 is a flowchart of the approach determination and a main subjectdetermination process of the second embodiment.

FIG. 9 is an explanatory diagram of recession determination of a thirdembodiment.

FIG. 10 is a flowchart of the recession determination and the mainsubject determination process of the third embodiment.

FIG. 11 is a block diagram of an imaging apparatus of a fourthembodiment.

FIG. 12 is an explanatory diagram of region division blocks of thefourth embodiment.

FIG. 13 is an explanatory diagram of approach determination of thefourth embodiment.

FIG. 14 is a flowchart of the approach determination and a main subjectdetermination process of the fourth embodiment.

FIG. 15 is a flowchart of a time correspondence process in each block ofthe fourth embodiment.

FIG. 16 is an explanatory diagram of the time correspondence process ineach block of the fourth embodiment.

FIG. 17 is a flowchart of a main subject setting process of the fourthembodiment.

FIG. 18 is an explanatory diagram of the main subject setting process ofthe fourth embodiment.

FIG. 19 is a block diagram when an embodiment is applied to a computerdevice.

DESCRIPTION OF EMBODIMENTS

Hereinafter, an embodiment will be described in the following order.

<1. Configuration of an image processing device of embodiment>

<2. Configuration of an imaging apparatus>

<3. Opportunity and objective of main subject determination, etc.>

<4. First embodiment of an imaging apparatus>

<5. Second embodiment of an imaging apparatus>

<6. Third embodiment of an imaging apparatus>

<7. Fourth embodiment of an imaging apparatus>

<8. Application to a program and a computer device>

<9. Modification examples>

1. Configuration of Image Processing Device of Embodiment

FIG. 1 shows a configuration example of an image processing deviceaccording to an embodiment.

An image processing device 1 includes a main subject determination unit2 and a subject distance change determination unit 3.

The subject distance change determination unit 3 detects a temporalchange of a distance from an imaging position with respect to a subjectpresent in an image and determines a tendency toward approach orrecession at the imaging position of the subject based on the detection.The tendency toward approach or recession is one of a tendency towardapproach of a subject to approach an imaging position or a tendencytoward recession of a subject to recede from an imaging position. Forexample, the subject distance change determination unit 3 detects adistance change of a subject in images from frame images sequentiallyinput as image data Dg, determines an tendency toward approach orrecession of the subject, and supplies sequentially the tendency towardapproach or recession to the main subject determination unit 2.Specifically, a distance change from the size of a subject image isobtained and the tendency toward approach or recession is determined.Alternatively, the subject distance change determination unit 3 inputsdistance information Id of the subject contained in the image data,obtains a distance change of the subject from the change in the distanceinformation Id, and determines the tendency toward approach orrecession.

The main subject determination unit 2 determines a main subject based onthe tendency toward approach or recession of each subject determined bythe subject distance change determination unit 3. For example, the mainsubject determination unit 2 determines the main subject based on adetermination result of the tendency toward approach. Alternatively, themain subject determination unit 2 determines the main subject based on adetermination result of a tendency toward recession. For example, someor all of the subjects having the tendency toward approach or therecession are determined as the main subjects.

Then, the main subject determination unit 2 outputs main subjectinformation Dm as a determination result.

The image processing device 1 including the main subject determinationunit 2 and the subject distance change determination unit 3 describedabove can be realized by a central processing unit (CPU) or a digitalsignal processor (DSP) serving as an arithmetic processing device.

It can also be considered that the function of the main subjectdetermination unit 2 is realized by a CPU or the like and the functionof the subject distance change determination unit 3 is realized by animage processing DSP or the like connected to the CPU or as acooperation process.

The flow of a process for the main subject determination performed bythe image processing device 1 is illustrated in FIG. 2A.

In Step F1, the subject distance change determination unit 3 detects thedistance change of the subject contained in the image data anddetermines the subject in which the tendency toward approach orrecession is shown.

In Step F2, the main subject determination unit 2 determines the subjectin which the tendency toward approach or recession has been observed asthe main subject.

In Step F3, the main subject determination unit 2 outputs the mainsubject information Dm which is the determination result of the mainsubject to transmit and receive the main subject information Dm to andfrom an application program or the like.

The application program or the like performs a process according to thefact that the main subject image is specified. For example, focuscontrol, a tracking process, or an image effect process is performed.

In FIGS. 2B and 2C, two schemes are shown as specific examples of theprocesses of Steps F1 and F2 in FIG. 2A.

The process example of FIG. 2B is a process of detecting an image sizeof the subject in the image data in each frame and obtaining its sizechange amount to determine the tendency toward approach or recession ofeach subject. That is, a temporal change of a distance of the subject isdetected as a size change on an image. The process of this idea isreferred to as a “size determination scheme” for description. First tothird embodiments to be described below will be described as examples inwhich the idea of the size determination scheme is used.

The process example of FIG. 2C is a process of dividing the image datainto a plurality of regions and detecting a temporal change of adistance of the subject in each division region (block) in an image. Theprocess of this idea is referred to as a “block determination scheme”for description. A fourth embodiment to be described is as an example inwhich the idea of the block determination scheme is used.

The size determination scheme in FIG. 2B will be described. In thiscase, the processes of Steps F10, F11, F12, and F13 are performed asStep F1 of FIG. 2A and Step F14 is performed as Step F2 of FIG. 2A.

In Step F10, the subject distance change determination unit 3 detects acandidate image which can become the main subject in the image data. Thecandidate image is, for example, a human face image, a human body image,a dog image, or a cat image. The subject distance change determinationunit 3 sets one candidate image or a plurality of candidate images suchas face images as the subjects present in an image through an imageanalysis process on the image data.

In Step F11, the subject distance change determination unit 3 calculatesthe size (area) of each candidate image. For example, the size iscalculated using, for example, the width×height of a region of thecandidate image or the number of pixels of the candidate image.

In Step F12, the subject distance change determination unit 3 calculatesa size change of each candidate image. For example, a difference betweenthe size of a candidate image of a current processing target frame incontinuous pieces of frame image data and the size of the candidateimage in the frame image data before a unit time (for example, one framebefore) is calculated. The size of the candidate image on the imagecorresponds to a distance between an imaging position (which is aposition of the imaging apparatus with which a cameraman performsimaging) and a subject considered as the candidate image. Accordingly,when the subject that is a candidate image is approaching, the sizeincreases with the progress of the frames. When the subject that is acandidate image is receding, the size decreases with the progress of theframes. That is, the size change can be comprehended as a distancechange.

In Step F13, the subject distance change determination unit 3 determinesthe tendency toward approach or recession of each candidate image. Forexample, when the size change is observed during a span of some time andthe subject approaches as the candidate image, the size of the subjectgradually increases. That is, the size change amount in a size expansiondirection is observed on average, cumulatively, or continuously to someextent. In this case, the candidate image can be determined to have thetendency toward approach at the imaging position.

In contrast, when the subject recedes as the candidate image, the sizeof the subject gradually decreases. That is, the size change amount in asize contraction direction is observed on average, cumulatively, orcontinuously to some extent. In this case, the candidate image can bedetermined to have the tendency toward recession at the imagingposition.

In Step F14, the main subject determination unit 2 selects the candidateimage having the tendency toward approach or the recession and sets thesubject of the candidate image as the main subject.

Through the above-described process, the subject approaching the imagingposition can be automatically determined to be the main subject or, incontrast, the subject receding from the imaging position can beautomatically determined to be the main subject.

Next, the block determination scheme of FIG. 2C will be described. Inthis case, the processes of Steps F20, F21, and F22 are performed asStep F1 of FIG. 2A and the processes of Steps F23 and F24 are performedas Step F2 of FIG. 2A.

In Step F20, the subject distance change determination unit 3 detects asubject distance of each of the division regions (blocks) which are aplurality of regions divided from the image data. As will be describedin detail below, the division region (block) refers to one of theregions which are a plurality of regions divided from one image, asindicated by the dotted lines of FIG. 12B. In Step F20, distanceinformation of the subject depicted in the division region is detectedin each division region.

In Step F21, the subject distance change determination unit 3 calculatesthe distance change in regard to each division region. For example, adifference of the subject distance between each division region of thecurrent processing target frame in continuous frame image data and eachdivision region in the frame image data before the unit time (forexample, one frame before) is calculated. Thus, the distance change ofthe subject in each division region is calculated.

In Step F22, the subject distance change determination unit 3 determinesthe tendency toward approach or recession of each division region. Forexample, when the distance change is observed during a span of some timeand the subject of the division region approaches, the value of thedistance gradually decreases and the distance change amount in adirection in which the distance is shortened is observed on average,cumulatively, or continuously to some extent. In this case, the subjectof the division region can be determined to have the tendency towardapproach at the imaging position.

In contrast, when the subject of the division region recedes, the valueof the distance gradually increases and the distance change amount in adirection in which the distance is lengthened is observed on average,cumulatively, or continuously to some extent. In this case, the subjectof the division region can be determined to have the tendency towardrecession at the imaging position.

In Step F23, the main subject determination unit 2 determines a subjectimage region containing the division region in which there is thetendency toward approach or the recession. The division regions aredivided from an image region and do not correspond to regions of asubject image in a one-to-one manner. For example, one subject image ispresent over the plurality of division regions in many cases. Thus, themain subject determination unit 2 determines a region range of onesubject image under conditions of, for example, a region (an adjacentregion or a region also adjacent to the adjacent region) which hassubstantially the same value of the subject distance as that of thedivision region having the tendency toward approach or the recession andis continuous with this division region.

Then, in Step F24, the subject image in the determined region range isdetermined to be a main subject.

Through the above-described process, the subject approaching the imagingposition can be automatically determined to be the main subject or, incontrast, the subject receding from the imaging position can beautomatically determined to be the main subject.

In the image processing device 1 in FIG. 1, the main subjectdetermination is performed as in each of the above-described examples,so that the subject intended as a target by the user can be estimated bya motion (approaching/receding) of the subject. Thus, the main subjectdetermination can be performed automatically without dependency on amanual manipulation of the user. Accordingly, the image processingdevice 1 in FIG. 1 is mounted on any of the various electronicapparatuses performing an operation according to the setting of the mainsubject, thereby considerably improving the operability of the user.

2. Configuration of Imaging Apparatus

Hereinafter, the main subject determination process will be described indetail exemplifying the imaging apparatus 10 including theabove-described image processing device.

A configuration example of the imaging apparatus 10 according to theembodiment is shown in FIG. 3. The configuration example of the imagingapparatus 10 is appropriate for a first embodiment. Configurationexamples of the imaging apparatus 10 according to second to fourthembodiments will be described at each time.

The imaging apparatus 10 is considered as a so-called digital stillcamera or a so-called digital video camera and is an apparatusperforming imaging or recording a still image or a moving image andincluding the image processing device described in the claims.

The imaging apparatus 10 shown in FIG. 3 has an optical system 11, animager 12, an optical system drive unit 13, a sensor unit 14, arecording unit 15, a communication unit 16, a digital signal processingunit 20, a control unit 30, a user interface controller (hereinafter,“UI controller”) 32, and a user interface 33.

The optical system 11 has lenses such as a cover lens, a zoom lens, anda focus lens and a throttle mechanism. By this optical system 11, lightfrom a subject is collected in the imager 12.

The imager 12, for example, has a CCD (Charge Coupled Device) type orCMOS (Complementary Metal OxideSemiconductor) type image sensor.

The imager 12, for example, performs a CDS (Correlated Double Sampling)process, an AGC (Automatic Gain Control) process and the like for anelectrical signal obtained through photoelectric conversion in the imagesensor, and further performs an A-D (Analog-Digital) conversion process.Then, the imager 12 outputs an imaging signal as digital data to thedigital signal processing unit 20 of a rear stage.

The optical system drive unit 13 drives the focus lens of the opticalsystem 11 and performs a focus operation under the control of thecontrol unit 30. Furthermore, the optical system drive unit 13 drivesthe throttle mechanism of the optical system 11 and performs exposureadjustment under the control of the control unit 30. Moreover, theoptical system drive unit 13 drives the zoom lens of the optical system11 and performs a zoom operation under the control of the control unit30.

The digital signal processing unit 20, for example, is configured as animage processor by a DSP and the like. The digital signal processingunit 20 performs various types of signal processes for a digital signal(captured image signal) from the imager 12.

For example, the digital signal processing unit 20 includes apre-processing unit 21, a synchronization unit 22, a YC generation unit23, a resolution conversion unit 24, a codec unit 25, and a candidatedetection unit 27.

The pre-processing unit 21 performs a clamping process of clamping ablack level of R, G, and B to a predetermined level, or a correctionprocess among color channels of R, G, and B with respect to the capturedimage signal from the imager 12.

The synchronization unit 22 performs a demosaicing process such thatimage data for each pixel has color components of all of R, G, and B.

The YC generation unit 23 generates (separates) a luminance (Y) signaland a color (C) signal from the image data of R, G, and B.

The resolution conversion unit 24 executes a resolution conversionprocess on the image data on which various types of signal processeshave been performed.

The codec unit 25, for example, performs a coding process for recordingor communication with respect to the image data for which the resolutionconversion has been performed.

The candidate detection unit 27 performs an image analysis process inunits of frames (or per intermittent frame) for a captured image signal(luminance signal and color signal) obtained by, for example, the YCgeneration unit 23, and then extracts a candidate image. In other words,face image detection, human body detection, and the like are performedfor image data continuously input on a time axis, and then imagesserving as candidates for a main subject are extracted.

Note that face detection, human body detection, and the like can beperformed using techniques of pattern matching and the like in imageanalysis performed with respect to captured image data, but if only adictionary used in pattern matching is replaced, other detectors canalso be realized in principle. For example, extraction of candidateimages of a main subject for dog detection (of a certain breed), catdetection, and the like is possible.

In addition, for example, detecting a moving body and setting the movingbody to be a candidate image using a technique of moving body detectionbased on a frame difference can also be considered, and a technique ofextracting an area of interest that is called saliency (Saliency) may beused.

Various techniques of extracting and selecting candidate images areconsidered.

The candidate detection unit 27, for example, a face image is detected,and an area in which the face image is present is extracted as acandidate image frame.

With regard to the extracted candidate image, position information ofthe candidate image frame (x and y coordinate values on a screen,information of a subject distance, and the like) or size information(for example, the width, height, and number of pixels of the candidateimage frame, and the like) are transferred to the control unit 30 ascandidate image information. Note that, here, since the candidate imageinformation is information indicating a frame of an image area servingas a candidate image, the term of candidate image information is alsoreferred to as “candidate image frame information.”

In addition, the candidate detection unit 27 may perform a smoothingprocess, an outlier (outlier) removal process, or the like for the imageto generate candidate image frame information.

A functional configuration in which the candidate detection unit 27 isimplemented in the digital signal processing unit 20 is set in theexample of FIG. 3, but this is an example, and the control unit 30 mayexecute the process of the candidate detection unit 27.

The control unit 30 is configured by a micro-computer (arithmeticprocessing device) that has a CPU, a ROM (Read Only Memory), a RAM(Random Access Memory), a flash memory, and the like.

By executing a program stored in the ROM, the flash memory, and thelike, the CPU comprehensively controls the entire imaging apparatus 10.

The RAM serving as a work area when the CPU performs various kinds ofdata processes is used for temporarily storing data, programs, and thelike.

The ROM and the flash memory (non-volatile memory) are used for storingan OS (Operating System) necessary for control of each unit by the CPU,content files such as image files, application programs for variousoperations, firmware, and the like. For example, programs for executingthe main subject determination process that will be described later inthe present example, and further application programs that use mainsubject determination results, and the like are stored therein.

The control unit 30 described above controls operations of necessaryunits relating to instruction of various signal processes in the digitalsignal processing unit 20, imaging operations and recording operationsaccording to a user manipulation, a reproducing operation of recordedimage files, camera operations such as zooming, focusing, and exposureadjustment, user interface operations, and the like.

In the case of the embodiment, the control unit 30 has the functions ofa main subject determination unit 30 a and a distance change calculationunit 30 b and performs approach determination (or recessiondetermination) and the main subject determination process described inthe first to fourth embodiments to be described below.

In the case of the configuration in FIG. 3, the distance changecalculation unit 30 b calculates the image size of the candidate imageset in the candidate detection unit 27, calculates a change in the imagesize at each unit time, and determines the tendency toward approach orthe recession from the calculation result.

The main subject determination unit 30 a performs a process of settingthe main subject in the candidate image based on the determinationresult of the distance change calculation unit 30 b.

The user interface 33 executes display output and audio output to auser, and receives input of a user manipulation. For this reason, theuser interface has a display device, a manipulation device, a speakerdevice, a microphone device, and the like. Herein, a display unit 34 anda manipulation unit 35 are shown.

The display unit 34 is a display unit that performs various types ofdisplay for a user (a photographer and the like), and for example, isformed using a display device, such as an LCD (Liquid Crystal Display)or an organic EL (Electro-Luminescence) display, which is formed on ahousing of the imaging apparatus 10. In addition, the display unit maybe formed using the LCD or the organic EL display in a so-called viewfinder form.

This display unit 34 includes the display device and a display driverthat allows the display device to perform display. The display driverallows various types of display to be performed on the display devicebased on the instruction of the control unit 30. For example, thedisplay driver reproduces and displays a still image or a dynamic imagecaptured and recorded in a recording medium, or displays a through image(subject monitoring image) as a dynamic image based on captured imagedata of each frame, which is captured during release (a shuttermanipulation) standby, on a screen of the display device. Furthermore,the display driver allows various manipulation menus, icons, messagesand the like, that is, a GUI (Graphical User Interface), to be displayedon the screen. In the case of the present embodiment, for example,display that helps the user to ascertain a determination result obtainedfrom main subject determination on the through image or reproduced imageis also performed.

The manipulation unit 35 has an input function of receiving a usermanipulation, and sends a signal corresponding to the input manipulationto the control unit 30.

The manipulation unit 35, for example, is realized by variousmanipulators provided on the housing of the imaging apparatus 10, atouch panel formed on the display unit 34, and the like.

As the manipulator on the housing, a reproduction menu start button, adecision button, an arrow key, a cancellation button, a zoom key, aslide key, a shutter button (a release button) and the like areprovided.

Furthermore, by a touch panel manipulation using the icons and menusdisplayed on the touch panel and the display unit 34, various kinds ofmanipulation may be possible.

An operation of the display unit 34 of the user interface 33 and thelike is controlled by the UI controller 32 according to instructions ofthe control unit 30. In addition, information of manipulation by themanipulation unit 35 is transmitted to the control unit 30 by the UIcontroller 32.

The recording unit 15 includes, for example, a non-volatile memory, andserves as a storage area for storing image files (content files) such asstill image data or dynamic image data, attribute information of theimage files, thumbnail images and the like.

The image files are stored in the form of, for example, a JPEG (JointPhotographic Experts Group), a TIFF (Tagged Image File Format), a GIF(Graphics Interchange Format) and the like.

The actual form of the recording unit 15 can be variously considered.For example, the recording unit 15 may be a flash memory that isembedded in the imaging apparatus 10, or may have a form based on amemory card (for example, a portable flash memory) attachable to anddetachable from the imaging apparatus 10 and a card recording andreproduction unit that performs recording and reproduction access to thememory card. Furthermore, the recording unit may be realized in the formof an HDD (Hard Disk Drive) and the like that is embedded in the imagingapparatus 10.

Furthermore, in the present example, a program for performing approachdetermination (or recession determination) and the main subjectdetermination process that will be described later may be stored in therecording unit 15.

The communication unit 16 performs data communication or networkcommunication with an external device in a wired or wireless manner.

For example, the communication unit performs communication of thecaptured image data (still image files or dynamic image files) with anexternal display apparatus, a recording apparatus, a reproductionapparatus and the like.

Furthermore, as a network communication unit, the communication unit mayperform communication through various networks, for example, theInternet, a home network, or a LAN (Local Area Network), or performvarious types of data transmission and reception with a server, aterminal and the like on the network.

The sensor unit 14 comprehensively represents various sensors. Forexample, a gyro sensor (angular velocity sensor), an accelerationsensor, and the like for detecting overall motions of the imagingapparatus 10 such as camera shake, attitudes and movements (panningmovement, tilting movement, and the like) of the imaging apparatus 10are provided.

In addition, a luminance sensor that detects external luminance forexposure adjustment and the like and a distance measuring sensor thatmeasures subject distances may be provided.

In addition, there are cases in which a zoom lens position sensor thatdetects positions of the zoom lens in the optical system 11 and a focuslens position sensor that detects positions of the focus lens areprovided as the sensor unit 14.

In addition, there are cases in which a sensor that detects openingamounts of a mechanical iris (diaphragm mechanism) is provided as thesensor unit 14.

The various sensors of the sensor unit 14 each transmit detectedinformation to the control unit 30. The control unit 30 can performvarious kinds of control using the information detected by the sensorunit 14.

In the imaging apparatus 10, the configuration portion of the imageprocessing device 1 described in FIG. 1 is as follows.

The configuration of the image processing device 1 in FIG. 1corresponding to the main subject determination unit 2 is implemented asthe main subject determination unit 30 a in the control unit 30 of theimaging apparatus 10 by software. The configuration corresponding to thesubject distance change determination unit 3 is implemented as thedistance change calculation unit 30 b in the candidate detection unit 27and the control unit 30 of the imaging apparatus 10 by software.

The control unit 30 controls the execution of an operation as an imageprocessing method described in the claims by executing a process basedon a program described in the claims.

3. Opportunity and Objective of Main Subject Determination, Etc.

In the present embodiment, main subject determination is performed asdescribed in the first to the fourth embodiments to be described later,and an opportunity, objective, etc. for performing main subjectdetermination in the imaging apparatus 10 will be described.

First, an example of using a main subject determination result will bedescribed.

Main subject determination is executed when, for example, a user(photographer) aims at a shutter timing (release timing), but thecontrol unit 30 can perform the following process after a main subjectis automatically determined.

Tracking Process

A main subject set in each captured frame is tracked. For example, amain subject is specified on a through image display for the user, andprovided for adjusting an angle of view performed by the user (forexample, for decision of a subject in a state in which a camera is heldin a hand).

Note that, as presentation of a main subject, highlight display of theframe of the main subject on the through image display by the displayunit 34 is considered. In addition, the highlight display or the likemay be performed for a given period immediately after the determination,or may be performed as long as the main subject is present within thethrough image.

Focusing

Auto focus is controlled for a main subject. In addition, in accordancewith the tracking process, focus is adjusted tracking the main subjecteven when the main subject moves around.

Exposure Adjustment Automatic exposure adjustment is performed based onbrightness (luminance) of a main subject.

Directivity adjustment

When sound collection is performed using a microphone together withcapturing (for example, dynamic image capturing), directivity adjustmentis performed according to a direction of a main subject within a fieldof view.

Zoom Control

Auto zoom is controlled with respect to a main subject. For example, thezoom lens is automatically driven so that the main subject is projectedin a captured image in a predetermined size or greater at all times.Together with the tracking process, an angle of view may be set to beadjusted using zoom according to a change in a distance to the mainsubject.

Recording Start Control

Main subject determination may be set to trigger a start of dynamicimage capturing. For example, dynamic image capturing and recording arestarted according to decision of a main subject.

In addition, a main subject can also be used in various signal processesperformed with respect to a captured image signal.

Image Effect Process

Image processes including image quality adjustment, noise reduction,skin color adjustment, and the like are performed only on the area of amain subject in each captured frame.

Alternatively, adding an image effect, for example, a mosaicing process,an airbrushing process, a paint-out process, or the like in an areaother than the area of the main subject is also considered.

Image Editing Process

An editing process such as framing, cropping, or the like is performedon a captured image or a recorded image.

For example, a process of cropping, enlarging, or the like of a partialarea within a frame in which a main subject is included can beperformed.

In addition, cutting of image peripheral portions of captured image dataor the like can be performed so that a main subject is disposed at thecenter of the image, and composition adjustment can be performed.

These are merely examples, and various processes of application programsor automatic adjustment functions included in the imaging apparatus touse a set main subject are considered in addition to them.

Next, at what point of time the main subject determination processshould be executed is also variously considered.

For example, whenever the imaging apparatus 10 is turned on to performimaging (a period in which the display unit 34 displays a throughimage), the main subject determination process may be performed.

In addition, when a main subject is determined and then the trackingprocess is performed, the main subject determination process may beperformed again at the time when the tracking is lost.

In addition, the main subject determination process may be set to startthrough user manipulation.

In addition, the process may be executed whenever a user selects adetermination execution mode, or tracking is lost.

In addition, automatically activating the main subject determinationprocess regardless of user manipulation may also be considered.

The following effects are achieved by performing the main subjectdetermination process.

As previously described, it is naturally difficult to perform amanipulation of designating a main subject when a photographer aims atthe subject holding the imaging apparatus 10.

Particularly, when an imaging direction is continuously changed in afixed direction as in panning and tilting, or when an imaging directionis changed not in a fixed direction, the manipulation of designating amain subject is difficult.

In addition, the action of designating a main subject many times isbothersome.

If main subject determination is set to be automatically performed, suchtroubles are overcome, and the effect of reduced stress on the user isobtained.

In addition, the imaging apparatus 10 that is carried and used by auser, such as a digital still camera, a camera included in a mobiletelephone, or the like used by general users, has the display unit 34 ofa small size, and thus it is difficult for the user to perform anaccurate manipulation of designating a main subject on a screen. Theproblem of erroneous designation is resolved by performing automaticdetermination as described in the present embodiment.

In addition, if the imaging apparatus 10 is set to automatically performmain subject determination, a user executes the main subjectdetermination in a situation in which he or she holds the imagingapparatus 10 to aim at a subject, or changes an imaging direction totrack the subject, and thus effects of enhancement of a feeling ofintelligence of the apparatus that the user senses from the apparatusand enhancement of added value are obtained.

In addition, since the user can use the imaging apparatus 10 with afeeling of being able to image a main figure only by naturally holdingthe apparatus, imaging opportunities and use cases associated therewithcan increase, and accordingly, a user-friendly camera can be provided tothe user.

Based on the above points, as a camera of hand-held type, the imagingapparatus 10 of the present embodiment that automatically performs mainsubject determination is particularly preferred.

4. First Embodiment of Imaging Apparatus

In the first embodiment, a main subject determination process in theimaging apparatus having the above-described configuration in FIG. 3will be described. This embodiment is an example in which the mainsubject determination is performed with the idea of the sizedetermination scheme described in FIG. 2B. The first embodiment isassumed to be an example in which an approaching subject is determinedto be the main subject.

FIG. 4A schematically shows a candidate image frame extraction operationperformed by the candidate detection unit 27.

FIG. 4A shows frames FR1, FR2, FR3, . . . of captured image signalsinput to the digital signal processing unit 20 through an operation ofthe optical system 11 and the imager 12 of the imaging apparatus 10. Thecandidate detection unit 27 performs detection of candidate images fromthe continuous frames (or intermittent frames) sequentially input asabove.

For example, as shown in the drawing, when three people are present inthe frame FR1, portions of face image portions are extracted ascandidate images and candidate image frame information of candidateimage frames E1, E2, and E3 is output. For example, the candidate imageframe information of the candidate image frame E1 is, for example,position information (x, y position information) in an image of thecandidate image frame E1 and the width w and the height h of thecandidate image frame. The x,y position information is, for example,coordinate values with the origin (0, 0) in the upper left corner of animage. The width w and the height h may be, for example, the number ofpixels. The candidate detection unit 27 generates such candidate imageframe information of the candidate image frames E2 and E3, and thentransmits or receives the candidate image frame information to or fromthe control unit 30.

In the same manner, the candidate detection unit 27 extracts candidateimages of the following frames FR2, FR3, . . . , generates candidateimage frame information for each of candidate image frames of theimages, and then transfers the information to the control unit 30.

The control unit 30 calculates a frame area (h×w) as the size of eachcandidate image frame whenever the candidate image frame information foreach frame is taken, and further detects a difference from the size ofthis candidate image of a previous frame as a change amount of the framearea. Then, the control unit 30 determines whether each candidate imagehas the tendency toward approach by observing a change in the differenceon a time axis.

For example, FIG. 4B shows an example of a captured image when a subjectperson considered to be the candidate image frame E1 is approaching inthe direction of the imaging apparatus 1. In a certain frame FRx, theframe area of the candidate image frame E1 is relatively small. However,in a frame FRy at a time point at which some time has passed, the framearea of the candidate image frame E1 increases. Whether the subject isapproaching is determined by observing the change in the size of thecandidate image frame.

FIG. 5A shows a change in the calculated frame area when the candidateimage frames E1, E2, and E3 are assumed to be continuously present inthe frames (FR1, FR2, . . . ) during a certain period.

For example, a case in which the photographer catches subjects of threepeople with the imaging apparatus 10 during a certain period is assumed.When each of the three people is moving or the photographer is movingcarrying the imaging apparatus 10, frame area distances of thecalculated candidate image frames E1, E2, and E3 are changed on the timeaxis. In this example, the frame area of the candidate image frame E1gradually increases. However, the frame areas of the candidate imageframes E2 and E3 are slightly changed, but are not considerably changedon average.

FIG. 5B shows a transition of the change amounts of the frame areas ofthe candidate image frames E1, E2, and E3. For example, a value at eachtime point is shown as a difference obtained by subtracting the framearea of a previous frame from the frame area of a current frame.Accordingly, when a relative distance to the imaging apparatus 10becomes closer than at the previous time point, the value of the areachange amount increases. When the relative distance is not changed, thearea change amount is 0. When the subject recedes, the area changeamount is a negative value.

Accordingly, a determination result of each area change amount is firstobtained using a distance determination threshold value Thd fordetermining whether the subject is approaching based on the area changeamount. When the area change amount exceeds the distance determinationthreshold value Thd, the subject is determined to approach by a distanceequal to or greater than a predetermined distance at this time point.

FIG. 5C shows the determination results of the candidate image framesE1, E2, and E3. When the area change amount exceeds the distancedetermination threshold value Thd, a flag of an “H” level is set.

As understood from the drawing, the value of the area change amount isnormally a high value and an E1 determination result is continuously setto “1” in the candidate image frame E1.

In the candidate image frame E2, an E2 determination result is set to“1” during only a certain period since the value of the area changeamount is a high value in some cases.

In the candidate image frame E3, an E3 determination result iscontinuously set to “0” since the value of the area change amount isnormally a low value.

In this example, the subject that is approaching on average,cumulatively, or continuously for some time is determined to be asubject with the tendency toward approach. The subject for which theperiod in which the determination result in FIG. 5C is “1” is long isdetermined to be a subject with the tendency toward approach. Forexample, when a period length of a continuous period, a cumulativeperiod, or the like in which the determination result is “1” is counted,the tendency toward approach can be determined.

For example, the candidate image frame E1 can be determined to have thetendency toward approach during this determination period since thedetermination result is “1” and the period is long.

The candidate image frame E2 can be said to be, for example, a subjecttemporarily approaching or receding. The candidate image frame E3 is asubject remaining at a relative distant position.

A period from determination start to determination end differs accordingto a specific process example. In a process example of FIG. 6 to bedescribed below, when a period length for which the determination resultis “1” is counted and the period length reaches a predetermined time,the candidate image frame is determined to have the tendency towardapproach. In this case, for example, when there is a subject for whichthe determination result is normally “1” as in the candidate image frameE1, a timing of the determination end is quickened. That is, thedetermination period length changes depending on a situation of thedetermination process. As another example, the period from thedetermination start to the determination end is set as a fixed periodlength.

A specific example of the approach determination and the main subjectdetermination process by the control unit 30 will be described withreference to FIG. 6. A process to be described below is a processperformed by the functions of the distance change calculation unit 30 band the main subject determination unit 30 a of the control unit 30.

When the control unit 30 starts the approach determination and the mainsubject determination process, first, the variable TSF=0 is set asinitial setting in Step F100 and a count value Cnt(n)=0 is set.

The variable TSF is a flag indicating whether the setting of the mainsubject ends. TSF=“0” indicates a non-determination state of the mainsubject.

The count value Cnt(n) is the value of a counter for determining a timelength in regard to a determination result obtained by comparing theabove-described area change amount and the distance determinationthreshold value Thd.

Note that “n” indicates natural numbers of 1, 2, 3 . . . and the countvalue Cnt (n) is set to be a count value corresponding to each detectedcandidate image frame E (n) like the candidate image frames E1, E2, andE3. When the three candidate image frames E1, E2, and E3 are detected,Cnt1, Cnt2, and Cnt3 are used as count values. For the sake ofdescription of the flowchart, it is advised that a process with regardto the count value Cnt(n) be understood as, for example, a processtargeting each of Cnt1, Cnt2, and Cnt3.

A frame area Area(n) to be described in FIG. 6 also similarly indicatesa frame area of each candidate image frame. A process for the frame areaArea(n) is used to mean a process on each of frame areas Area1, Area2,Area3, . . . of the candidate image frames E1, E2, and E3, . . . , forexample.

Further, an area change amount Diff(n) similarly indicates an areachange amount of each candidate image frame. The process for the areachange amount Diff(n) is used to mean a process on each of area changeamounts Diff1, Diff2, Diff3, . . . of the candidate image frames E1, E2,and E3, . . . , for example.

The same also applies to a determination flag Flg(n) and an offset valueOFS(n).

In addition, a candidate image frame E(n) indicates each of thecandidate image frames E1, E2, E3 . . . , but it is preferablydistinguished for each subject over a plurality of frames. In an examplein which the candidate detection unit 27 extracts a face, when a personA, a person B, and a person C are subjects, for example, the face imageportion of the person A is set to be the candidate image frame E1, theface image portion of the person B to be the candidate image frame E2,and the face image portion of the person C to be the candidate imageframe E3 common in each of the frames. If only a person D is interposedas a subject in a certain middle frame, the face image portion of theperson D is set to be a candidate image frame E4. Thus, it is better forthe candidate detection unit 27 to not only merely detect a “face” butalso determine an entity (individual).

For example, the control unit 30 performs the processes of Steps F101 toF116 at a timing of every one frame.

In Step F101, the control unit 30 takes candidate image frameinformation of a certain frame from the candidate detection unit 27. Forexample, with regard to each candidate image frame E(n), informationincluding an x value and a y value of two-dimensional (x-y) coordinatevalues of image data as position information, and a width w and a heighth of the candidate image frame as size information are acquired.

In Step F102, the control unit 30 calculates the frame area Area(n) ofeach frame image frame E(n). The frame area Area(n)=width w×height h isobtained. Here, the frame is assumed to be square. However, thecandidate image frame E(n) may not necessarily be square, but can alsobe considered to be circular, elliptical, or amorphous. In this case,the frame area Area(n) may be assumed to be the number of pixelscontained in the candidate image frame E(n).

Next, in Step F103, the control unit 30 obtains an area change amountDiff(n) of each candidate image frame E(n). The area change amountDiff(n) is obtained as a difference from a previous frame. That is,“area change amount Diff(n)=Area(n)−Area(n)pre” is obtained. Area(n)preis a frame area Area(n) of the previous frame of the candidate imageframe E(n). For example, the frame area Area(n) obtained in Step F102when an image one frame before is a target is considered to be“Area(n)pre” at a process time point of the current frame.

In Step F104, the control unit 30 confirms a variable TSF. When thevariable TSF=0 is set, the process proceeds to Step F105.

For example, during a period in which the main subject determination isperformed and the main subject determination process is not necessarydue to operation mode setting or the like, the variable TSF=1 is set,and thus the main subject determination process is not performed.

For example, when the main subject determination process of FIG. 6 isconfigured to be performed as necessary through a user's manipulation orautomatic activation determination, the process of Step F104 may beunnecessary.

In Steps F105, F106, and F107, the control unit 30 confirms whether thearea change amount Diff(n) of each candidate image frame E(n) indicatesapproach.

That is, whether the area change amount Diff(n) is a change amountindicating the approach of a predetermined amount or more is determinedusing the distance determination threshold value Thd described in FIG.5B.

Therefore, in Step F105, the control unit 30 compares the area changeamount Diff(n) of each candidate image frame E(n) to the distancedetermination threshold value Thd. When “Diff(n)>Thd” is satisfied, thedetermination flag Flg(n)=1 (approaching) is set in Step F106. When“Diff(n)>Thd” is not satisfied, the determination flag Flg(n)=0 (notapproaching) is set in Step F107.

The determination flag Flg(n) corresponds to the determination result of“1” or “0” described in FIG. 5C.

Subsequently, in Steps F108, F109, and F110, the control unit 30 setsthe offset value OFS(n) for a counting process depending on whether thedetermination flag Flg(n) is “1” or “0.”

For example, the control unit 30 sets the offset value OFS(n)=a in StepF109 when the determination flag Flg(n)=1 is set. Here, a is apredetermined value for incrementing the counter. For example, α=1 isset.

The control unit 30 sets the offset value OFS(n)=3 in Step F110 when thedetermination flag Flg(n)=0 is set. Here, β is a predetermined value forkeeping a count value or decrementing a count value. When the countvalue is kept, β=0 is set. When the counter is decremented, for example,P=−1 is set.

Then, in Step F111, the control unit 30 performs a process of the countvalue CNT(n) of the counter counting a time length in which approach isobserved. Specifically, CNT(n)=CNT(n)+OFS(n) is set.

When α=1 is considered and the offset value OFS(n) is set in theforegoing Step F109, the count value CNT(n) is incremented.

When β=0 (or β=−1) is considered and the offset value OFS(n) is set inStep F110, the count value CNT(n) is kept (or is decremented).

The count value CNT(n) is incremented when the approach is detected.Therefore, when the subject of the candidate image frame E(n) isapproaching, the count value becomes a value corresponding to the lengthof the detected period. When the approach is not detected (for example,a situation in which a subject person stops) and the count value CNT(n)is configured to be kept, the count value CNT(n) is a cumulative valueof the approach detection and thus increases according to the approachdespite the fact that the subject person stops. When the approach is notdetected and the count value CNT(n) is configured to be decremented, thecount value CNT(n) increases according to the average approach. However,when the approach is not detected, the count value decreases. Thus, whenthe subject temporarily stops or recedes, subsequent determination ofthe tendency toward approach may be disadvantageous.

Then, in Step F112, the control unit 30 determines whether eachcandidate image frame E(n) has the tendency toward approach.Specifically, the control unit 30 determines whether the count valueCNT(n) indicating the time length of the approach detection becomes avalue equal to or greater than a predetermined time threshold value Tht.

When none of the count values Cnt(n) of the candidate image frames E(n)reaches the time threshold value Tht, the variable TSF=0 remains in StepF113, the end of the determination is not determined in Step F116, andthe process returns to Step F101. In this case, the process after StepF101 is performed, as described above, based on the candidate imageframe information input for a subsequent frame.

In Step F116, when the variable TSF=0 is set, the main subjectdetermination is assumed not to be completed and the determinationprocess continues. When the variable TSF=1 is set, the main subjectdetermination is assumed to be completed. When the variable TSF=1 isdetected in the above-described Step F104, the determination endsdirectly.

Although the detailed description is omitted, for example, the user maybe allowed to select the main subject through a manipulation of touchingthe main subject on the screen of the display unit 34 or a manipulationof half pressing the shutter button according to a predeterminedposition of a subject on the screen, in parallel to the automatic mainsubject determination of this example. When the user performs such adesignation manipulation during the execution of the process of FIG. 6,it is desirable to prefer the user's manipulation. Thus, when the mainsubject setting is performed through such a manual manipulation, thevariable TSF=1 is set. In this case, the process of FIG. 6 may end(interruption end) through the determination of Steps F104 and F116.

The main subject determination performed through the approachdetermination is performed during a certain time length. Therefore, whenthe process is not performed on the candidate image frame informationfor some time (the number of frames), the determination does not end inStep F116, as described above, the process returns to Step F101, and theprocess is repeated.

Here, for example, as shown in FIG. 5, the certain candidate image frameE1 may be discontinuous, but there is a situation in which the approachis detected in a plurality of frames. Then, there are many opportunitiesin which the count value Cnt1 of the candidate image frame E1 isincremented in Step F111 as time passes, and thus the count value Cnt1advances earlier than the count values Cnt2 and Cnt3.

Then, at a certain time point, the count value Cnt1 first reaches thetime threshold value Tht.

In this case, the control unit 30 causes the process to proceed fromStep F112 to Step F114.

In Step F114, the control unit 30 determines that the candidate imageframe E(n) for which the count value Cnt(n) reaches the time thresholdvalue Tht is the main subject and performs the setting of the mainsubject. Then, the variable TSF=1 is set. That is, the subject of thecandidate image approaching on average during a certain period isdetermined to be a subject having the “tendency toward approach” and isset as the main subject.

Then, in Step F115, the main subject information is output to betransmitted to or received from, for example, an application program ora control program using the main subject information.

In this case, the end of the determination is considered in Step F116.That is, for example, the candidate image frame E1 is set as the mainsubject, and thus the approach determination and the main subjectdetermination process of FIG. 6 are completed.

In this process example, the process continues until the variable TSF=1is set. However, actually, it is appropriate to provide a predeterminedrestriction time limit. That is, when the main subject may not bedetermined despite the fact that a predetermined time has passed fromthe process start time point of FIG. 6, the process is configured to endas a process in which there is no main subject.

The approach determination and the main subject determination processare performed, as described above.

In the approach determination and the main subject determinationprocess, the tendency toward approach is determined in the plurality offrames among the extracted candidate images.

For example, a subject of high certainty at which a photographer holdinga camera is considered to aim as a target has the tendency towardapproach in many cases. For example, when a camera is oriented to achild, the child is observed to approach in many cases. Accordingly,when automatically setting an approaching subject as the main subject,the process is appropriate for the photographer, thereby considerablyimproving operability at the time of imaging. Even a user unaccustomedto an imaging manipulation can capture a still image or a moving imagewith high quality, for example, when focus control or exposure controlis automatically performed on the main subject through the main subjectdetermination.

Other various specific process examples can be considered as theapproach determination and the main subject determination process.

For example, when β=0 is set in Step F110 of FIG. 6, the count valueCNT(n) corresponds to a cumulative value of the number of times theapproach detection is cumulatively performed.

When f=−1 is set in Step F110 of FIG. 6, the count value CNT(n) can beregarded as a value at which the approach detection is performed onaverage.

That is, in this case, the approach determination of Step F112 is thedetermination in which the subject is approaching cumulatively or onaverage.

On the other hand, when the subject is continuously approaching during aperiod equal to or greater than a predetermined period, the subject maybe determined to have the tendency toward approach. For such occasions,the count value CNT(n) is incremented when the determination flagFlg(n)=1 is considered in Step F108, and the count value CNT(n) may bereset to zero when the determination flag Flg(n)=0 is considered in StepF108. Then, when the state of the determination flag Flg(n)=1continuously reaches a predetermined time, the count value Cnt(n)reaches the time threshold value Tht in Step F112 and the candidateimage frame E(n) is determined to have the “tendency toward approach”and the candidate image frame E(n) is determined to be the main subject.

According to the setting of the continuously approaching subject as themain subject with the tendency toward approach, for example, a stoppingor temporarily receding subject can be prevented from being determinedto be the main subject. Accordingly, this determination is appropriatefor the main subject determination to be performed prudently.Conversely, when the determination of the main subject is desired to beperformed in a short time or the main subject is desired to be set aseasily as possible, the subject that is approaching on average orcumulatively is appropriately determined to have the tendency towardapproach.

In the cases of the cumulative determination, the average determination,and the continuous determination described above, it is appropriate thatspecific determination values corresponding to the time threshold valueTht are different.

For example, when the approach is detected in the continuous,cumulative, or average sense under the predetermined time condition, thesubject is determined to have the tendency toward approach, but weightof the determination may be changed temporarily.

For example, the values α and β substituted into the offset value OFS(n)set in Steps F109 and F110 of FIG. 6 are changed with the progress ofthe process.

In general, when a photographer holds a camera, the photographer firstdoes not particularly decide a subject set as a main figure in manycases. For example, the photographer holds the camera to search for animaging target within a range of vision. In this case, when thephotographer finds a gradually approaching person and attempts tophotograph the person, the photographer performs an operation ofgradually adjusting the person at the center in a subject direction.That is, there is a probability that the subject observed to approach inan image is thought to be a subject that the photographer desires todecide as the main subject as time passes. Accordingly, during theexecution of the approach determination and the main subjectdetermination process, the weight of the approach detection becomesstronger as time passes. Specifically, the value a substituted into theoffset value OFS(n) in Step F109 increases step by step. By doing so,there is a high probability of the main subject determination conformingto the thought of the photographer being performed.

In the example of FIG. 6, at the time point at which the count valueCNT(n) of the certain candidate image frame E(n) is equal to or greaterthan the time threshold value Tht, the main subject determination isperformed and the determination ends, and thus the determination periodis indefinite. On the other hand, a process example of setting a givendetermination period and setting, as the target of Step F114, thecandidate image frame E(n) for which the count value CNT(n) is thelargest or one or a plurality of candidate image frames E(n) for whichthe count value CNT(n) is equal to or greater than a predetermined valuewithin the determination period to determine the candidate image frameto be the main subject can also be considered.

Another condition (AND condition) may be considered to be added to thedetermination of the tendency toward approach in order to determine asubject as the main subject. For example, the fact that a subjectdistance is less than a predetermined distance, the fact that a subjectimage has a size equal to or greater than a predetermined size, the factthat a type of image is specific, or the fact that the center of asubject image with the tendency toward approach is within apredetermined distance (a distance on the x and y coordinates) from thecenter point of a frame image may also be added to the condition.

By adding such conditions, it is possible to increase a probability of asubject that the photographer thinks to image more being determined tobe the main subject as well as merely having the tendency towardapproach. Accordingly, the user may be configured to select theadditional conditions.

5. Second Embodiment of an Imaging Apparatus

A main subject determination process will be described according to thesecond embodiment. As in the first embodiment, this embodiment is anexample in which an approaching subject is determined to be the mainsubject with the idea of the size determination scheme described in FIG.2B.

In the case of the second embodiment, the configuration of an imagingapparatus 10 is shown in FIG. 7. In FIG. 7, the same reference numeralsare given to the same portions as those of FIG. 3 and the descriptionthereof will be omitted.

The imaging apparatus shown in FIG. 7 is different from that in FIG. 3in that an attribute identification unit 28 is provided in the digitalsignal processing unit 20.

For example, the attribute identification unit 28 identifies anattribute of a subject that is detected as a candidate image throughimage analysis by the candidate detection unit 27, and then transmits orreceives attribute information to or from the control unit 30. Theattribute information is, for example, information indicating whetherthe subject detected as the candidate image is an adult or a child orinformation indicating whether the subject is a male or a female. Forexample, the attribute identification unit 28 discriminatescharacteristic points of a face image and determines gender or ages.

For example, the attribute identification unit 28 can be functionallyconfigured to be performed by the digital signal processing unit 20. Anexample in which the attribute identification unit 28 is realized as aprocessing function in the control unit 30 can also be considered.

FIG. 8 shows a process example of the control unit 30 (a main subjectdetermination unit 30 a and a distance change calculation unit 30 b).The same step numbers are given to the same processes of FIG. 6described above and the repeated description will be omitted.Differences from FIG. 6 are processes of Steps F101A and F112A.

For example, the control unit 30 performs the processes of Steps F101 toF116 at a timing of every one frame. In Step F101A, the control unit 30takes candidate image frame information of a certain frame from thecandidate detection unit 27. For example, with regard to each candidateimage frame E(n), information including an x value and a y value oftwo-dimensional (x-y) coordinate values of image data as positioninformation, and a width w and a height h of the candidate image frameare acquired as position information. The control unit 30 furtheracquires attribute information AT(n) of each candidate image frame E(n)from the attribute identification unit 28. The attribute information is,for example, identification information for discriminating an adult froma child or for discriminating a male from a female. Specifically, whenthe candidate image frames E1, E2, and E3 are present, pieces ofattribute information AT1, AT2, and AT3 are acquired. For example, theattribute information AT1 is information regarding “child and female,”the attribute information AT2 is information regarding “adult and male,”and the attribute information AT3 is information regarding “adult andfemale.”

In Steps F102 to F111, a process of determining the tendency towardapproach is performed on each candidate image frame E(n), as in FIG. 6.

In the case of FIG. 8, in Step F112, a time threshold value fordetermining a subject to have the tendency toward approach is assumed tobe a time threshold value Tht(ATn) corresponding to the attributeinformation. The time threshold value Tht(ATn) means that a valueserving as the time threshold value is switched according to theattribute information.

For example, the time threshold value Tht is set to a relatively smallvalue in the cases of “child and female” and “child and male,” is set toa relatively large value in the case of adult and female, and is set toa larger value in the case of “adult and male.”

The control unit 30 may store threshold values according to theattribute information as table data in an internal storage unit orthreshold values according to the attribute information may be decidedby setting numeral values on a program. As a coefficient value in regardto a fixed threshold value Tht, a coefficient value may be set for eachattribute. The control unit 30 may obtain the threshold value Tht(ATn)according to the attribute information in Step F112A.

In Step F112A, for each candidate image frame E(n), the count valueCNT(n) is compared to the time threshold value Tht(ATn), and thuswhether having the tendency toward approach or not is consequentlydetermined. Then, the candidate image having the tendency towardapproach is determined to be the main subject in Step F114.

According to the process example of FIG. 8, ease of determining thecandidate image to be the main subject differs according to theattribute information. For example, when the time threshold value Tht(ATn) is a small value in the case of a child as described above, it iseasy to determine the child to be the main subject. When the timethreshold value Tht(ATn) is smaller in the case of a female than in thecase of a male, it is easy to determine the female to be the mainsubject.

Thus, the ease of determining the candidate image to be the main subjector difficulty in determining the candidate image to be the main subjectcan be adjusted according to the attribute of the subject. Accordingly,it is easy to determine the candidate image desired by the user to bethe main subject. For example, for a user mainly images a child,accuracy of the main subject determination is improved and thedetermination time is shortened by the above-described time thresholdvalue.

In particular, the user may set a preference order according to varioususe cases. For example, when a child is imaged, setting is performed sothat the child is preferred. When a male is imaged, setting is performedso that the male is preferred. Thus, when the user can select the timethreshold value according to the attribute, the main subjectdetermination is realized quickly and with high accuracy according to animaging purpose of the user.

Various pieces of attribute information can be further considered. Forexample, attribute information obtained by estimating ages in greaterdetail than the attribute information of adult and child from an imageand identifying age groups (child/youth/middle age/old age) may be used.

An example in which a personal face recognition scheme is introduced,children, family, and acquaintances of a user are determined, the timethreshold value is lowered in this case, and such a candidate image iseasily determined to be the main subject can also be considered. Forexample, the faces of the children, family, or the like of the user areregistered in advance and characteristic data are obtained in advance.When the attribute identification unit 28 detects a face conforming tothe characteristic data, attribute information is generated asinformation regarding a close relative and the control unit 30 sets thetime threshold value Tht(ATn) to a small value in this case.

The attribute information is not limited to a person, but attributeinformation of an animal such as a dog or a cat or a type of animal maybe generated and the time threshold value may be changed.

In the process of the second embodiment, the same modification examplesas those of the first embodiment can be considered.

6. Third Embodiment of Imaging Apparatus

A main subject determination process will be described according to thethird embodiment. As in the first embodiment, this embodiment is anexample in which a receding subject is determined to be the main subjectwith the idea of the size determination scheme described in FIG. 2B.

As in FIGS. 5A, 5B, and 5C described above, FIGS. 9A, 9B, and 9C showthe determination result using a change in the calculated frame area, anarea change amount, and a distance determination threshold Thd when thecandidate image frames E1, E2, and E3 are assumed to be continuouslypresent in the frames (FR1, FR2, . . . ) during a certain period.

In the case shown in FIG. 9A, the frame area gradually decreases whenthe candidate image frame E1 is focused on. Here, unlike the definitionof the area change amount described in FIG. 5B, that is, “(area changeamount)=(frame area of current frame)−(frame area of previous frame),”the area change amount in FIG. 9B is defined as (area changeamount)=(frame area of previous frame)−(frame area of current frame).”Then, the value of the area change amount is a large value when theframe area decreases. For example, the area change amount is shown inFIG. 9B according to the frame area change in FIG. 9A. When the areachange amount is compared to the distance determination threshold valueThd and the determination is performed, the candidate image frames E1,E2, and E3 are shown in FIG. 9C.

In the case of this example, the determination result in the candidateimage frame E1 is continuously “1.”

In the candidate image frame E2, the determination result is “1” duringa period in which the area change amount is a high value.

In the candidate image frame E3, the E3 determination result iscontinuously “0” since the area change amount is normally a low value.

As in the first embodiment, when a period length of a continuous period,a cumulative period, or the like in which the determination result is“1” is counted, a subject that is receding on average or cumulatively,or continuously, that is, a subject with the tendency toward recession,can be determined. In the case of FIG. 9, the candidate image frame E1is determined to be a subject with the tendency toward recession sincethe subject is receding on average, cumulatively, or continuously forsome time.

FIG. 10 shows a process example of recession determination and the mainsubject determination process of the control unit 30 (the main subjectdetermination unit 30 a and the distance change calculation unit 30 b).The same step numbers are given to the same processes of FIG. 6described above and the repeated description will be omitted. Adifference from FIG. 6 is the process of Step F103B.

For example, the control unit 30 performs the processes of Steps F101 toF116 at a timing of every one frame. At this time, in Step F103B, thecontrol unit 30 defines an area change amount Diff(n) of each candidateimage frame E(n) as follows:area change amount Diff(n)=Area(n)pre−Area(n).

Area(n)pre is a frame area Area(n) of the previous frame of thecandidate image frame E(n). For example, the frame area Area(n) obtainedin Step F102 when an image one frame before is a target is considered tobe “Area(n)pre” at a process time point of the current frame.

The subsequent processes of Steps F104 to F116 are performed as in FIG.6.

When the area change amount Diff(n) is calculated in Step F103B, asdescribed above and the frame area is less than that of the previousframe (=the subject is receding), the value of the area change amountincreases.

Therefore, when area change amount Diff(n) is compared to the distancedetermination threshold value Thd and the determination flag Flg(n) isset in Steps F105 to F107 and the process for the count value Cnt(n) isperformed in Steps F108 to F111, the count value Cnt(n) reaches the timethreshold Tht at a certain time point in Step F112 in the candidateimage frame E(n) of the subject that is receding on average orcumulatively. In this case, the subject of the candidate image frameE(n) is assumed to have the tendency toward recession and is selected asthe main subject in Step F114.

In the recession determination and the main subject determinationprocess of FIG. 10, the tendency toward recession is determined in theplurality of frames among the extracted candidate images. Then, the mainsubject is determined based on the determination result of the tendencytoward recession.

For example, a photographer holding a camera aims at a receding subjectin many cases. When the photographer desires to photograph a subjectthat is moving at a high speed, such as a train, an airplane, or asubject in motor sports or desires to photograph a subject moving in adirection in which a wild bird, an animal, or the like is commonlyreceding from a person, the receding subject is automatically set as themain subject so that the process is appropriate for the photographer,thereby considerably improving operability at the time of imaging.

Even a user unaccustomed to an imaging manipulation can capture a stillimage or a moving image with high quality, for example, when focuscontrol or exposure control is automatically performed on the mainsubject through the main subject determination.

Other various specific process examples can be considered as therecession determination and the main subject determination process. Forexample, the description at the time of the approach determination inFIG. 6 has been made. However, when β=0 is set in Step F110 even in thecase of FIG. 10, the subject is determined to have the tendency towardrecession in a case in which a situation in which the subject iscumulatively receding is detected. When β=−1 is set in Step F110, thesubject is determined to have the tendency toward recession in a case inwhich a situation in which the subject is receding on average isdetected. When the determination flag Flg(n)=0 is considered in StepF108 and the count value CNT(n) is reset to zero, only the subject thatis continuously receding is determined to have the tendency towardrecession.

According to the determination of the continuously receding subject tobe the main subject with the tendency toward recession and setting thesubject as the main subject, this determination and setting isappropriate for the main subject determination to be performedprudently. Particularly a subject such as a train, an airplane, a wildbird, or an animal that is receding is continuously receding in manycases. When such a target is intended to be imaged, the accuracy of themain subject determination is improved through continuous determination.

On the other hand, when a subject that is not observed to continuouslymove is intended to be imaged and when the determination of the mainsubject is desired to be performed in a short time or the main subjectis desired to be set variously as easily as possible, the subject thatis receding on average or cumulatively is appropriately determined tohave the tendency toward recession.

In the cases of the cumulative determination, the average determination,and the continuous determination described above, it is appropriate thatspecific determination values corresponding to the time threshold valueTht are different.

For example, when the recession is detected in the continuous,cumulative, or average sense under the predetermined time condition, thesubject is determined to have the tendency toward recession, but weightof the determination may be changed temporarily. Specifically, the valuea substituted into the offset value OFS(n) set in Steps F109 isincreased or conversely decreased step by step.

Further, a process example of setting a given determination period andsetting, as the target of Step F114, the candidate image frame E(n) forwhich the count value CNT(n) is the largest or one or a plurality ofcandidate image frames E(n) for which the count value CNT(n) is equal toor greater than a predetermined value to determine the candidate imageframe to be the main subject within the determination period can also beconsidered.

Another condition may be considered to be added as an AND condition tothe determination of the tendency toward recession in order to determinea subject as the main subject. For example, the fact that a subjectdistance is equal to or greater a predetermined distance, the fact thata subject image has a size equal to or less than a predetermined size,the fact that a type of image is specific, or the fact that the centerof a subject image with the tendency toward recession is within apredetermined distance (a distance based on the x and y coordinates)from the center point of a frame image may also be added to thecondition.

By adding such conditions, it is possible to increase a probability of asubject that the photographer thinks to image more being determined tobe the main subject as well as merely having the tendency towardrecession. Accordingly, the user may be configured to select theadditional conditions.

The idea of the second embodiment may be applied even to thedetermination of the tendency toward recession and another timethreshold value Tht(ATn) may be used according to the attributeinformation.

7. Fourth Embodiment of Imaging Apparatus

A main subject determination process will be described according to thefourth embodiment. This embodiment is an example in which an approachingsubject is determined to be the main subject with the idea of the blockdetermination scheme described in FIG. 2C.

In the case of the second embodiment, the configuration of an imagingapparatus 10 is shown in FIG. 11. In FIG. 11, the same referencenumerals are given to the same portions as those of FIG. 3 and thedescription thereof will be omitted.

The imaging apparatus 10 in FIG. 11 includes a distance sensor 17 andhas the function of a distance calculation unit 29 in the digital signalprocessing unit 20.

The distance sensor 17 detects a subject distance in an imagingdirection. The distance calculation unit 29 calculates the subjectdistance for each of the division regions (block) divided from acaptured image using a value detected by the distance sensor 17.

A known technology such as a phase difference sensor scheme, theTime-of-Flight scheme, or the Kinet scheme can be used as a subjectdistance detection technology by the distance sensor 17 and the distancecalculation unit 29. For example, the phase difference sensor scheme isa scheme of obtaining a distance of a target subject from a distance(the number of pixels) between the pixels in which the same targetsubject is detected in captured images of the imager disposed on theright and left sides. The Time-of-Flight scheme is a scheme in which thedistance sensor 17 emits and receives infrared light and divides a timetaken to reflect the emitted infrared light from a target subject andreceive the reflected light by the speed of the infrared light to obtaina distance.

FIG. 12 shows a distance detection operation for each block.

FIG. 12A shows frames FR1, FR2, FR3, . . . of captured image signalsinput to the digital signal processing unit 20 through an operation ofthe optical system 11 and the imager 12 of the imaging apparatus 10. Inparallel to the inputting of the images, the distance sensor 17 operatesto measure the distance of a subject and the detected information isinput to the distance calculation unit 29.

The distance calculation unit 29 calculates the subject distance foreach block in the frame image data FR1, FR2, FR3, . . . .

The functions of the main subject determination unit 30 a and thedistance change calculation unit 30 b are provided in the control unit30.

In the imaging apparatus 10, the configuration portion of the imageprocessing device 1 described in FIG. 1 is as follows.

A configuration corresponding to the main subject determination unit 2of the image processing device 1 in FIG. 1 is implemented as the mainsubject determination unit 30 a on the control unit 30 of the imagingapparatus 10 by software. A configuration corresponding to the subjectdistance change determination unit 3 is implemented as the distancechange calculation unit 30 b and the distance calculation unit 29 byhardware or software.

In the example of FIG. 12, the distance calculation unit 29 isfunctionally configured to be executed by the digital signal processingunit 20, but this is merely an example. The function of the distancecalculation unit 29 can be considered to be implemented on the controlunit 30 by software.

FIG. 12B shows an example of the blocks which are division regions. Onepiece of image data is divided into a plurality of blocks, as indicatedby a dotted line. As shown in the lower part of FIG. 12B, the blocks arereferred to as blocks BK (BK1, BK2, . . . BK(M)) for description.

The distance calculation unit 29 obtains the subject distance for eachof the blocks BK (BK1, BK2, . . . BK(M). In the lower port of FIG. 12B,the subjects distances of the blocks BK are exemplified (numericalvalues in meters or infinity ∞). For example, 20 m is exemplified forthe block BK4 and infinity ∞ is exemplified for the block BK3.

For example, the distance calculation unit 29 obtains the subjectdistance for each block BK in this way in each frame, and then transmitsor receives distance information of each block BK to and from thecontrol unit 30.

FIG. 13A shows a change in the subject distance calculated in each framein regard to each block BK. Here, the blocks BK1, BK2, and BK(x) areexemplified.

In this example, the distance of a subject pictured in the block BK(x)is gradually shortened. The distances of the subjects pictured in theblocks BK1 and BK2 are slightly changed, but are not considerablychanged on average.

The same subjects are not necessarily located at the same blocks on thecaptured image data. Here, during a period from the start of thedetermination to the end of the determination, the same subjects (forexample, person A, person C, and person C) are assumed to be pictured atthe blocks BK1, BK2, and BK(x). Actually, when a subject moves in thehorizontal direction with respect to the imaging apparatus 10 or aphotographer pans or tilts the imaging apparatus 10, the blocks in whicha certain subject is imprinted are changed for each frame. Because ofconsideration of a necessity of such a situation, a time matchingprocess is applied for each block in Step F202 in the process example ofFIG. 14 to be described.

Now, the description will continue assuming that person A, person B, andperson C are continuously pictured in the blocks BK1, BK2, and BK(x) ofFIG. 13A in each frame

FIG. 13B shows transition of the distance change amount of each of theblocks BK1, BK2, and BK(x). For example, a value at each time point isshown as a difference obtained by subtracting the distance in a previousframe from the distance in a current frame in regard to each block BK.Accordingly, when the relative distance of the subject present withinthe block BK with respect to the imaging apparatus 10 decreases morethan at a previous time point, the distance change amount is a smallvalue (negative value). When the relative distance is not changed, thedistance change amount is 0. When the relative distance increases, thedistance change amount is a large value.

Accordingly, a determination result of each block BK is obtained usingthe distance determination threshold value Thd used to determine whetherthe subject becomes closer is obtained by the distance change amount.When the distance change amount is less than the distance determinationthreshold value Thd, the subject of the block BK is determined toapproach by a distance equal to or greater than a predetermined distanceat that time point.

FIG. 13C shows the determination results of the blocks BK1, BK2, andBK(x). This is assumed to be a flag with an “H” level when the distancechange amount is less than the distance determination threshold valueThd.

As understood from the drawing, the determination result of the blockBK(x) is continuously “1.” For the block BK2, “1” is set during only acertain period so that temporary approach is indicated. Thedetermination result of the BK1 is continuously set to “0.”

In this example, a subject that is approaching on average, cumulatively,or continuously for some time is determined to be a subject with thetendency toward approach. A subject for which a period in which thedetermination result of FIG. 13C is “1” is long is determined to be thesubject with the tendency toward approach. For example, when a timelength of a continuous period, a cumulative period, or the like in whichthe determination result is “1” is counted, the tendency toward approachcan be determined.

For example, since the determination result of the block BK(x) is “1”and the period is long, a subject contained in the block BK(x) can bedetermined to have the tendency toward approach during thisdetermination period.

The block BK2 can be said to be, for example, a block in which a subjectthat is temporarily approaching or receding is contained. The block BK1is a block in which a subject that is kept at a relatively distantposition is contained.

A period from the start of the determination to the end of thedetermination differs according to a specific process example. In aprocess example of FIG. 14 to be described below, a period length inwhich the determination result is “1” is counted and a subject of theblock BK is determined to have the tendency toward approach when theperiod length reaches a predetermined time. In this case, for example,when there is a subject for which the determination result is normally“1” as in the block BK(x), a timing of the end of the determination isquickened. That is, the determination period length changes depending ona situation of the determination process. As another example, the periodfrom the determination start to the determination end is set as a fixedperiod length.

A specific example of the approach determination and the main subjectdetermination process by the control unit 30 will be described withreference to FIG. 14. A process to be described below is a processperformed by the functions of the distance change calculation unit 30 band the main subject determination unit 30 a of the control unit 30.

When the control unit 30 starts the approach determination and the mainsubject determination process, first, the variable TSF=0 (the mainsubject is not determined) which is a flag indicating whether thesetting of the main subject is completed is set and a count valuebCnt(m)=0 is set as initial setting in Step F200.

The count value bCnt(m) is the value of a counter for determining a timelength in regard to a determination result obtained by comparing theabove-described distance change amount and the distance determinationthreshold value Thd.

Note that “m” indicates natural numbers of 1, 2, 3 . . . and the countvalue bCnt (m) is set to be a count value corresponding to each block ofthe blocks BK1, BK2, BK3, . . . , BK(M). For the sake of the process,bCnt1, bCnt2, . . . , and bCnt(M) are used as count values in regard tothe blocks BK1 to BK(M). For the sake of description of the flowchart, aprocess with regard to the count value bCnt(m) is assumed to refer to aprocess performed on each of the count values bCnt1, bCnt2, . . . , andbCnt(M).

Db(m) described in the process of FIG. 14 indicates a distance detectedin regard to each block and bDiff(m) indicates a distance change amountof each block BK. A process with regard to the distance Db(m) and thedistance change amount bDiff(m) is used to mean a process for thedistance and the distance change amount with regard to the blocks BK1 toBK(M).

The same also applies to a determination flag Flg(m) and an offset valueOFS(m).

For example, the control unit 30 performs the processes of Steps F201 toF217 at a timing of every one frame.

In Step F201, the control unit 30 takes pieces of distance informationDb1 to Db(M) with regard to the respective blocks BK1 to BK(M) in acertain frame from the distance calculation unit 29. The information is,for example, information regarding the values of the distances shown inthe lower part of FIG. 12B.

The control unit 30 performs the time matching process on each block BKin Step F202 and obtains the distance change amount bDiff(m) for eachblock BK in Step F203. That is, the distance change amount bDiff(m) is adifference between a distance value of a current frame and a distancevalue of a previous frame.

That is, “distance change amount bDiff(m)=Db(m)−Db(m)pre” is obtained.Db(m)pre is a distance value of the corresponding block BK(m) in theprevious frame of a current processing target block BK(m).

Here, the time matching process of Step F202 will be described withreference to FIGS. 15 and 16. As described above, the same subjects arenot necessarily located at the same blocks on the captured image data ofeach frame when time passes. When a relative position between a subjectand the imaging apparatus 10 in the horizontal and vertical directionsis changed, for example, the subject located at the block BK4 in theimmediately previous frame is located at the block BK5 in the currentframe in some cases. In consideration of this case, a distancedifference from the previous frame of the block BK5 is output, thisdifference is not the distance change amount of the subject.Accordingly, matching (tracking) a subject with a block is performed inStep F202.

The description will be schematically made with reference to FIG. 16.FIG. 16 shows a case in which a position of a certain subject P in animage deviates between a current frame FRc and a previous frame FRpre.

In the current frame FRc, a face portion of the subject P is assumed tobe present in a block BK(y). However, in the previous frame FRpre, theface portion is not present in the same block BK(y)pre and is located ina block BK(z)pre. Since it is originally necessary to obtain a distancechange of the subject P, a distance change amount bDiff(y) desired to beobtained in the block BK(y) of the current frame FRc is a differencefrom the distance value of the block BK(z)pre of the previous frameFRpre in this case. Accordingly, the time matching process is performedso that the block BK(z)pre mentioned here is considered to be the blockof the previous frame with regard to the block BK(y).

The time matching process of Step F202 will be described with referenceto FIG. 15.

In Step F230 of FIG. 15, the control unit 30 first confirms distanceinformation of nine blocks centering on a block BK(m)pre of the previousframe FRpre corresponding to the target block BK(m) of the current frameFRc.

As for the block BK(y) in FIG. 16, the control unit 30 confirms nineblocks surrounded by a thick frame centering on the same block BK(y)preof the previous frame FRpre. As shown in the drawing, the nine blocksare referred to as BKx1 to BKx9. The distance values of the blocks BKx1to BKx9 are assumed to be “10,” “10,” “9,” “10,”, “7,” “6,” “9,” “7,”and “5” (units are, for example, meters). The distance value of theblock BK(y) of the current frame is assumed to be “5,” as shown in thedrawing.

When the distance value of each of the blocks BKx1 to BKx9 is confirmed,the control unit 30 calculates absolute distance differences bDiffX1 tobDiffX9 between the target block BK(m) of the current frame FRc and therespective corresponding blocks BKx1 to BKx9 of the previous frame FRprein Step F231 as follows.

bDiffX 1 = Db(m) − Db(BKx 1) bDiffX 2 = Db(m) − Db(BKx 2) …bDiffX 9 = Db(m) − Db(BKx 9)

Here, the distance Db(m) is a distance value of the target block BK(m)and is a distance value “5” of the block BK(y) in the example of FIG.16.

The distances Db(BKx1) to Db(BKx9) are distance values of the nineblocks BKx1 to BKx9 and are the above-described values “10,” “10,” “9,”“10,”, “7,” “6,” “9,” “7,” and “5” in the example of FIG. 16.

In Step F232, the control unit 30 determines the minimum value of theabsolute distance difference among bDiffX1 to bDiffX9. Then, in StepF233, the block in which the absolute distance difference is the minimumvalue is set to the block BK of the previous frame corresponding to thecurrent target block BK(m) and the distance Db of the block BK is set tothe distance Db(m)pre of the previous frame. Then, the calculation ofthe distance change amount “bDiff(m)=Db(m)−Db(m)pre” in Step F203 ofFIG. 14 is performed using the distance Db(m)pre.

For example, in the example of FIG. 16, since the block BKx9 is theblock in which the absolute distance difference is the minimum, theblock of the previous frame corresponding to the processing target blockBK(y) is the block BKx9 and the distance value Db(m)pre=5 is considered.

That is, in regard to the blocks of the current frame, the block inwhich the absolute distance difference is small is searched for near thesame blocks of the previous frame and the block is set as thecorresponding block (of the same subject) under the assumption that thesubject distances or the positions in the horizontal and verticaldirections are mostly not changed for the time of one frame.

In Step F202 of FIG. 14, the matching is performed on each block BK inthis way. Thus, the distance change amount bDiff(m) calculated in StepF203 in regard to each block BK(m) indicates a distance change of thesame subject.

In Step F204, the control unit 30 confirms the variable TSF. When thevariable TSF=0 is set, the process proceeds to Step F205. The meaning ofStep F204 is the same as that of Step F104 of FIG. 6.

In Steps F205, F206, and F207, the control unit 30 confirms whether thedistance change amount bDiff(m) of each block BK(m) indicates theapproach.

That is, whether the distance change amount bDiff(m) is a change amountindicating the approach of a predetermined amount or more is determinedusing the distance determination threshold value Thd, as described inFIG. 13B.

Therefore, in Step F205, the control unit 30 compares the distancechange amount bDiff(m) of each block BK to the distance determinationthreshold value Thd. When “bDiff(m)<Thd” is satisfied, the determinationflag Flg(m)=1 (approaching) is set in Step F206. When “bDiff(m)<Thd” isnot satisfied, the determination flag Flg(m)=0 (not approaching) is setin Step F207.

The determination flag Flg(n) corresponds to the determination result of“1” or “0” described in FIG. 13C.

Subsequently, in Steps F208, F209, and F210, the control unit 30 setsthe offset value OFS(m) for a counting process depending on whether thedetermination flag Flg(m) is “1” or “0.”

For example, the control unit 30 sets the offset value OFS(m)=α in StepF209 when the determination flag Flg(m)=1 is set. Here, α is apredetermined value for incrementing the counter. For example, α=1 isset.

The control unit 30 sets the offset value OFS(m)=β in Step F210 when thedetermination flag Flg(m)=0 is set. Here, β is a predetermined value forkeeping a count value or decrementing a count value. When the countvalue is kept, β=0 is set. When the counter is decremented, for example,β=−1 is set.

Then, in Step F211, the control unit 30 performs a process of the countvalue bCNT(m) of the counter counting a time length in which approach isobserved. Specifically, bCNT(n)=bCNT(m)+OFS(m) is set.

When α=1 is considered and the offset value OFS(m) is set in theforegoing Step F209, the count value bCNT(m) is incremented.

When β=0 (or β=−1) is considered and the offset value OFS(m) is set inStep F210, the count value bCNT(m) is kept (or is decremented).

The count value bCNT(m) is incremented when the approach is detected.Therefore, the count value is a value corresponding to the length of theperiod in which the subject of the block BK(m) is detected to approach.When the approach is not detected (for example, a situation in which asubject person stops) and the count value bCNT(m) is configured to bekept, the count value bCNT(m) is a cumulative value of the approachdetection. Therefore, when the approach is detected many times, thecount value bCnt(m) increases. When the approach is not detected and thecount value bCNT(m) is configured to decrease, the count value bCNT(m)increases according to the average approach. However, when the approachis not detected and the count value is decremented, the count valuebCNT(n) increases according to the average approach. However, when theapproach is not detected, the count value decreases. Thus, when thesubject temporarily stops or recedes, subsequent determination of thetendency toward approach may be disadvantageous.

Then, in Step F212, the control unit 30 determines whether each blockBK(m) has the tendency toward approach. Specifically, the control unit30 determines whether the count value bCNT(m) indicating the time lengthof the approach detection becomes a value equal to or greater than thepredetermined time threshold value Tht.

When none of the count values bCnt(m) of the blocks BK(m) reaches thetime threshold value Tht, the variable TSF=0 remains in Step F213, theend of the determination is not determined in Step F217, and the processreturns to Step F201. In this case, the process after Step F201 isperformed, as described above, based on distance information Db(m) ofeach block BK(m) input for a subsequent frame.

In Step F217, when the variable TSF=0 is set, the main subjectdetermination is assumed not to be completed and the determinationprocess continues. When the variable TSF=1 is set, the main subjectdetermination is assumed to be completed. This step is the same as StepF116 of FIG. 6. For example, even when the main subject is set through auser's manual manipulation performed in parallel, the process from StepF217 may end (interruption end).

The main subject determination performed through the approachdetermination is performed during a certain time length. Therefore, whenthe process is not performed on each block BK(m) for some time (thenumber of frames), the determination does not end in Step F217, theprocess returns to Step F201, and the process is repeated.

Here, for example, as shown in FIG. 13, the certain block BK(x) may bediscontinuous, but there is a situation in which the approach of asubject is detected in a plurality of frames. Then, there are manyopportunities in which the count value bCnt(x) of the block BK(x) isincremented in Step F211 as time passes, and thus the count valuebCnt(x) advances earlier than the count values bCnt1, bCnt2, and thelike of the other blocks. Then, at a certain time point, the count valuebCnt(x) first reaches the time threshold value Tht.

In this case, the control unit 30 causes the process to proceed fromStep F212 to Step F214.

In Step F214, the control unit 30 determines that the block BK(m) forwhich the count value bCnt(m) reaches the time threshold value Tht isthe block in the main subject is contained and performs the setting ofthe main subject. That is, the block in which the subject is approachingon average during a certain period is determined to be the block inwhich the subject has the “tendency toward approach” and setting of themain subject is performed. Then, in Step F215, the variable TSF=1 isset.

Then, in Step F216, the main subject information is output to betransmitted to or received from, for example, an application program ora control program using the main subject information.

In this case, the determination is considered to end in Step F217.

Here, the setting of the main subject will be described in Step F214. Inthe case of the process of FIG. 14, the determination of the tendencytoward approach in Step F212 is performed in units of the blocks BK andis not necessarily performed in units of subjects. Thus, in Step F214, asubject range is searched for based on the blocks and the subject rangeis determined to be the main subject.

The detailed description will be made with reference to FIGS. 17 and 18.

FIG. 18A shows a part of the image data of one frame. Dotted linesindicate the blocks BK.

Now, it is assumed that a subject P is approaching and a certain blockBK(p) is determined to have the tendency toward approach at a certaintime point in Step F212. Here, the block BK(p) is a part of the subjectP and it is not appropriate that only the portion of the block BK(p) isset as the main subject. Accordingly, to set the range of the subject Pas the main subject, the control unit 30 performs a process shown inFIG. 17 in Step F214.

First, in Step F240, a distance difference between the block BK(m)determined to have the tendency toward approach in Step F212 and each ofthe other blocks BK is calculated. That is, differences of the distancevalues between, for example, the block BK(p) in FIG. 18A which is theblock BK(m) and all of the other blocks BK1 to BK(M) (here, excludingthe block BK(p)) in the frame are calculated. Then, in Step F241, theblocks BK for which the distance difference is within a predeterminedvalue are extracted.

The blocks BK indicated by the diagonal lines in FIG. 18B are the blocksfor which the distance difference from the block BK(p) is within thepredetermined value.

Next, in Step F242, the control unit 30 groups the blocks continuouswith the block BK(m) (the block BK(p) in FIG. 18) determined to have thetendency toward approach among the blocks BK extracted in theabove-described manner. The continuous blocks refer to the blocks with arelation physically continuous on an image plane, such as the blocksadjacent to the block BK(p) and the blocks also adjacent to the adjacentblocks. For example, a block group MA shown in FIG. 18C is formed.

Then, in Step F243, the control unit 30 sets the formed block group MAas a region forming a main subject image, sets this image range as themain subject, and generates main subject information.

Consequently, the number of blocks grouped irrespective of the otherblocks in Step F242 is only one block in some cases. In this case, onlythe range of the block BK(m) determined to have the tendency towardapproach may be determined to be the main subject.

When the determination of the tendency toward approach in units ofblocks is achieved by performing such a process, the image region of acertain subject can be appropriately set to the main subject image basedon the blocks. This is because the differences of the distance valuesmostly do not occur when the subject is the same, and thus the range ofthe subject can be appropriately determined by extracting the blocks forwhich the distance differences are small and grouping the continuousblocks, as described above.

The region of the subject image set as the main subject may be a regionof the formed block groups or a region of a plurality of blocks may beset as a region of the main subject image in a shape similar to acircular shape, an elliptical shape, or a square shape.

Further, image analysis may be performed in the range of the formedblock group, a contour is determined through face detection, bodydetection, or the like, and the contour may be determined to be the mainsubject.

Another process may be performed to determine the main subject in StepF214 in addition to the above-described process. For example, as in thefirst embodiment, the candidate detection unit 27 may be provided in thedigital signal processing unit 20 or the control unit 30 and candidateimages may be extracted in parallel to the process of FIG. 14. In thiscase, a candidate image frame in which the block BK(m) determined tohave the tendency toward approach in Step F212 may be searched for and asubject of the candidate image frame may be set as the main subject.

In this way, the approach determination and the main subjectdetermination process are performed.

In the approach determination and the main subject determinationprocess, the tendency toward approach is determined in a plurality offrames for each of the blocks divided from an image. Then, the settingof the main subject is performed based on the determination of theblocks of the tendency toward approach. Thus, operability at the time ofimaging is considerably improved as in the first embodiment. A stillimage or a moving image with high quality can be captured, for example,when focus control or exposure control is automatically performed on themain subject through the main subject determination.

Other various specific process examples can be considered as theapproach determination and the main subject determination process. Forexample, as described at the time of the approach determination of FIG.6, even in the case of FIG. 14, when β=0 is set in Step F210, thetendency toward approach is determined in the case of the detection of asituation of cumulative approach. When β=−1 is set in Step F210, thetendency toward approach is determined in the case of the detection of asituation of average approach. When the determination flag Flg(n)=0 isconsidered in Step F208 and the count value bCNT(m) is reset to zero,only the block BK that is continuously approaching is determined to havethe tendency toward approach. In each case, the advantages described inthe first embodiment can be obtained.

For example, when the approach is detected in the continuous,cumulative, or average sense under a predetermined time condition, thesubject is determined to have the tendency toward approach, but weightof the determination may be changed temporarily. Specifically, the valueat substituted into the offset value OFS(m) set in Steps F209 areincreased or conversely decreased step by step.

Further, a process example of setting a given determination period andsetting, as the target of Step F214, the block BK(m) for which the countvalue bCNT(m) is the largest or one or a plurality of blocks BK(m) forwhich the count value bCNT(m) is equal to or greater than apredetermined value to determine the block BK to be the main subjectwithin the determination period can also be considered.

Another condition may be considered to be added as an AND condition tothe determination of the tendency toward approach in order to determinea subject as the main subject. For example, the fact that a subjectdistance of the block BK(m) is less than a predetermined distance, thefact that a subject contained in the block BK(m) is a specific type ofimage, or the fact that the block BK(m) is within a predetermineddistance (a distance based on the x and y coordinates) from the centerpoint of a frame image may also be added to the condition.

All of the blocks BK1 to BK(M) have been set as the targets and theprocess has been performed. However, for example, some of the blocks BKsuch as only the blocks BK near the middle portion of the image data maybe set as the targets and the process of FIG. 14 may be performed. Forexample, when a photographer is considered to adjust an imagingdirection to catch a subject that the photographer thinks to set as themain subject in the middle portion as much as possible, only the blocksBK near the middle portion can be set as targets and the process of FIG.14 can be performed, thereby improving the accuracy of the setting ofthe main subject. Further, the processing load of the control unit 30can be reduced.

The time matching process described in FIGS. 15 and 16 has beenperformed in Step F202 of FIG. 14, but a process example in which thetime matching process is not performed can also be considered.

If the time matching process on each block is not performed, the blocksBK and a subject correspond with each other continuously during adetermination period only when a certain subject comes straight from thefront to approach in the direction of the imaging apparatus 10. Forexample, when a subject approaching in an oblique direction, a situationin which the subject is contained in other blocks BK as a frameprogresses occurs.

Conversely, if the time matching process on each block is not performed,only a subject approaching naturally from the front can be selected asthe main subject. That is, a case in which the tendency toward approachin the front direction is set as a determination condition of the mainsubject may be a process example in which Step F202 of FIG. 14 is notperformed.

The division of the blocks BK can be considered in various manners.

In the above-described embodiments, the blocks BK1 to BK(M) have beendescribed. However, various division numbers such as division into 4, 6,8, 9, . . . , 30, and 40 can be considered.

As the number of blocks is large, the processing burden of the approachdetermination increases. However, even when a zoom state or a subject issmall, the main subject determination can be performed with highcertainty. In contrast, as the number of blocks is small, the processingburden decreases.

All of each block BK to be divided do not have to have the same sizes orthe same area shape. It is also considered that, for example, one blockBK of the end parts of an image is set as a wide range and the centerpart of the screen is divided into fine block BKs. Considering that amain subject is highly likely to be positioned in the periphery of thecenter of the screen, dividing the center of the screen into small areasis appropriate.

8. Application to a Program and a Computer Device

Hereinabove, the embodiments of the image processing device 1 and theimaging apparatus 10 have been described, and the above-described mainsubject determination process can be executed by hardware, or bysoftware.

A program of an embodiment is a program that causes an arithmeticprocessing device, for example, a CPU (Central Processing Unit), a DSP(Digital Signal Processor), or the like, to execute the processes shownin the above-described embodiments.

That is, the program is a program causing an arithmetic processingdevice to perform a process of detecting a temporal change of a distancefrom an imaging position in regard to a subject present in an image anddetermining the tendency toward approach or recession of a subject withrespect to the imaging position based on the detection and a process ofdetermining the main subject based on the determined tendency towardapproach or recession of each subject.

To be specific, the program of the embodiment may be a program thatcauses the arithmetic processing device to execute the process shown inFIG. 2, FIG. 6, FIG. 8, FIG. 10 or FIG. 14.

Using such a program, a device that executes the above-described mainsubject determination can be realized using the arithmetic processingdevice.

Such a program can be recorded in advance on an HDD as a recordingmedium embedded in an appliance such as a computer device, a ROM in amicrocomputer having a CPU, and the like.

Alternatively, the program can be temporarily or permanently stored(recorded) in a removable recording medium such as a flexible disk, aCD-ROM (Compact Disc Read Only Memory), a MO (Magnet optical) disc, aDVD (Digital Versatile Disc), a Blu-ray Disc (registered trademark), amagnetic disk, a semiconductor memory, or a memory card. Such aremovable recording medium can be provided as so-called packagesoftware.

Furthermore, such a program can be downloaded from a download sitethrough a network such as a LAN (Local Area Network) or the Internet, inaddition to the installation from the removable recording medium to apersonal computer and the like.

Furthermore, such a program is suitable for the image processing deviceof the embodiment to be extensively provided. For example, the programis downloaded to a personal computer, a portable information processingapparatus, a cellular phone, a game device, a video player, a PDA(Personal Digital Assistant) and the like, so that the portableinformation processing device and the like are available as the imageprocessing device according to an embodiment of the present disclosure.

For example, in the computer device as illustrated in FIG. 19, the sameprocess as the main subject determination process can be performed inthe image processing device 1 of FIG. 1, and the imaging apparatus 10.

In FIG. 19, a CPU 71 of a computer device 70 performs various processesaccording to a program stored in a ROM 72 or a program loaded from astorage unit 78 to a RAM 73. Furthermore, the RAM 73 appropriatelystores data and the like which are necessary when the CPU 71 performsthe various processes.

The CPU 71, the ROM 72, and the RAM 73 are connected to one anotherthrough a bus 74. Furthermore, an input and output interface 75 is alsoconnected to the bus 74.

The input and output interface 75 is connected to an input unit 76including a keyboard, a mouse and the like, an output unit 77 includinga display such as a CRT (Cathode Ray Tube), an LCD, or an organic ELpanel, and a speaker, the storage unit 78 including a hard disk, and acommunication unit 79 including a modem and the like. The communicationunit 79 performs a communication process through a network including theInternet.

Furthermore, a drive 80 is connected to the input and output interface75 according to necessity, a removable medium 81 such as a magneticdisk, an optical disc, a magneto optical disc, or a semiconductor memoryis appropriately mounted thereon, and a computer program read from thedrive is installed in the storage unit 78 according to necessity.

When the aforementioned main subject determination process is performedby software, a program constituting the software is installed from anetwork or a recording medium.

The recording medium, for example, as illustrated in FIG. 19, isconfigured by the removable medium 81 including a magnetic disk(including a flexible disk), an optical disc (including a Blu-ray disc,a CD-ROM, and a DVD, a magneto optical disc (including a MD (MiniDisc)), a semiconductor memory and the like which are distributed todeliver a program to a user, separately from an apparatus body with theprogram recorded therein. Alternatively, the recording medium is alsoconfigured by the ROM 72, a hard disk included in the storage unit 78,and the like, which are delivered to a user in the state of beingincorporated in advance into the apparatus body with the programrecorded therein.

In the computer device 70, when dynamic image data is input through areception operation by the communication unit 79, or a reproductionoperation in the drive 80 (removable medium 81) or the recording unit78, the CPU 71 executes the functions of the subject distance changedetermination unit 3 and the main subject determination unit 2 describedabove based on the program. In other words, by executing the process ofFIGS. 2, 6, 8, 10, and 14, a main subject can be determinedautomatically regarding the input image data.

9. Modification Examples

Various modification examples of the above-described embodiments can beconsidered.

For example, in the third embodiment, the example in which the tendencytoward recession is determined with the idea of the size determinationscheme described in FIG. 2B has been described. However, even in thecase of the block determination scheme described in FIG. 2C, thetendency toward recession can also be determined and the main subjectcan be determined among receding subjects (blocks BK). For example, StepF203 of the process in FIG. 14 may be modified such that the distancechange amount “bDiff(m)=Db(m)pre-Db(m)” is calculated. By doing so, theblock of the tendency toward recession can be determined even in thecase of the block determination scheme.

The various processes have been described in the determination of thetendency toward approach and the determination of the tendency towardrecession, but an approach determination mode and a recessiondetermination mode may be switched according to a subject targeted bythe user. For example, when a person such as a child is photographed,the approach determination mode is selected. Then, the process of thefirst embodiment is performed. On the other hand, when a train, anairplane, a wild bird, or the like is desired to be photographed, therecession determination mode is selected. Then, for example, the processof the third embodiment is performed.

The approach determination mode and the recession determination mode maybe automatically switched according to a camera mode (a landscape mode,a portrait mode, or the like).

The approach determination and the recession determination can also beconsidered to be performed simultaneously in parallel. In this case, forexample, the main subject is considered to be determined at a time pointat which a corresponding subject is searched for in either the tendencytoward approach or the recession.

The setting of the time threshold value Tht and the distancedetermination threshold value Thd described in each embodiment can beappropriately modified according to a product which is the imagingapparatus 10 or the image processing device 1, a use form, or the like.The user may be configured to set any desired values.

By setting the time threshold value Tht or the distance determinationthreshold value Thd, a criterion of the main subject image (thecandidate image frame E(n) or the block BK(m)) can be modified. Bysetting the time threshold value Tht, swift determination can beselected and prioritized or high certainty determination can be selectedand prioritized.

In addition, the main subject information has been described as alsobeing used in an image effect process and an image editing process,however, it is also preferable to perform the main subject determinationprocess targeting a reproduced image to this end.

The main subject determination process targeting a reproduced image isof course assumed in the image processing device of FIG. 1, theinformation processing device of FIG. 19, and the like.

In addition, a result of the main subject determination process may beadded to still image data or dynamic image data imaged and recordedthereafter as metadata. In other words, information representing a mainsubject is added to a still image file, or the like.

In addition, manipulation of designating a main subject throughmanipulation by a photographer may be set to be possible while a throughimage is displayed and, at the same time, the main subject determinationprocess is performed.

Furthermore, the process of determining a main subject has beendescribed mainly on the assumption of capturing still images in theembodiments, however, the process of the embodiments described above canbe applied as a process of performing main subject determination on aplurality of captured frames during standby for capturing a dynamicimage, and capturing and execution of recording of a dynamic image.

Additionally, the present technology may also be configured as below.

(1)

An image processing device including:

a subject distance change determination unit configured to detect atemporal change of a distance from an imaging position to each subjectpresent in an image and determine a tendency toward approach orrecession of the each subject with respect to the imaging position onthe basis of the detection; and

a main subject determination unit configured to determine a main subjecton the basis of the tendency toward approach or recession of the eachsubject determined by the subject distance change determination unit.

(2)

The image processing device according to (1),

wherein the subject distance change determination unit determines thetendency toward approach of the each subject to the imaging position onthe basis of information regarding the temporal change of the distanceof the each subject, and

wherein the main subject determination unit determines the main subjecton the basis of a determination result of the tendency toward approach.

(3)

The image processing device according to (1) or (2), wherein the subjectdistance change determination unit detects a temporal change of a sizeof a subject image in the image as the temporal change of the distance.

(4)

The image processing device according to any one of (1) to (3), whereinthe subject distance change determination unit detects, as the temporalchange of the distance, a temporal change of a size of a subject imagein the image, the subject image being one or more of candidate imagesextracted in the image.

(5)

The image processing device according to any one of (1) to (4),

wherein the subject distance change determination unit determines thatthe subject that is approaching on average, cumulatively, orcontinuously is a subject with the tendency toward approach, as adetection result of the temporal change of the distance, and

wherein the main subject determination unit determines a part or all ofthe subject determined to have the tendency toward approach as the mainsubject.

(6)

The image processing device according to any one of (1) to (5), furtherincluding:

an attribute identification unit configured to identify an attribute ofthe subject and output attribute information,

wherein the subject distance change determination unit changes adetermination condition of the tendency toward approach of the subjectaccording to the attribute information.

(7)

The image processing device according to (1) or (2), wherein the subjectdistance change determination unit detects the temporal change of thedistance of the subject in each division region in the image.

(8)

The image processing device according to (7), wherein the subjectdistance change determination unit determines a division region at whichthe subject of the division region of a current processing target imageis located in an image before a unit time and detects the temporalchange of the distance of the subject as a difference between a subjectdistance of the determined division region and a subject distance of thedivision region of the current processing target image.

(9)

The image processing device according to claim (7) or (8),

wherein the subject distance change determination unit determines thatthe subject of the division region that is approaching on average,cumulatively, or continuously is the subject with the tendency towardapproach, as a detection result of the temporal change of the distance,and

wherein the main subject determination unit determines the main subjecton the basis of information regarding the division region in which thesubject is determined to have the tendency toward approach.

(10)

The image processing device according to (9), wherein the main subjectdetermination unit groups different division region in which a samesubject as the subject contained in one division region in which thesubject is determined to have the tendency toward approach is contained,and sets an image range serving as the main subject based on a range ofthe grouped one or more of division regions.

(11)

The image processing device according to (10), wherein the main subjectdetermination unit groups the one division region and the differentdivision region for which a distance difference of a subject distance iswithin a predetermined value and which is a region continuous with theone division region.

(12)

The image processing device according to (1),

wherein the subject distance change determination unit determines thetendency toward recession of the each subject from the imaging positionon the basis of information regarding the temporal change of thedistance of the each subject, and

wherein the main subject determination unit determines the main subjecton the basis of a determination result of the tendency toward recession.

REFERENCE SIGNS LIST

-   1 image processing device-   2 main subject determination unit-   3 local motion detection unit-   4 global motion detection unit-   10 imaging apparatus-   11 optical system-   12 imager-   13 optical system drive unit-   14 sensor unit-   15 recording unit-   16 communication unit-   17 distance sensor-   20 digital signal processing unit-   21 pre-processing unit-   22 synchronization unit-   23 YC generation unit-   24 resolution conversion unit-   25 codec unit-   27 candidate detection unit-   28 attribute identification unit-   29 distance calculation unit-   30 control unit-   30 a main subject determination unit-   30 b distance change determination unit-   32 UI controller-   33 user interface-   34 display unit-   35 manipulation unit-   70 computer device-   71 CPU

The invention claimed is:
 1. A hand-held camera comprising: an imagesensor configured to capture an image; a subject distance changedetermination circuitry configured to detect a temporal change of adistance from an imaging position to each subject of a plurality ofsubjects present in the image, and determine a tendency toward recessionof the each subject of the plurality of subjects with respect to theimaging position on a basis of the temporal change of the distance fromthe imaging position to the each subject; a main subject selectioncircuitry configured to select a first main subject on a basis of thetendency toward recession of the each subject of the plurality ofsubjects that has been determined by the subject distance changedetermination circuitry, wherein the first main subject has a tendencytowards recession; and a control circuitry configured to track the firstmain subject in response to the main subject selection circuitryselecting the first main subject.
 2. The hand-held camera according toclaim 1, wherein the control circuitry is further configured todetermine whether the first main subject is being tracked when the imageis captured by the image sensor, and control a memory to store a portionof the image that is captured as a still image in response todetermining that the first main subject is being tracked when the imageis captured.
 3. The hand-held camera according to claim 1, wherein thesubject distance change determination circuitry is further configured todetermine a tendency toward approach of the each subject to the imagingposition on a basis of information regarding the temporal change of thedistance of the each subject, and wherein the main subject selectioncircuitry is further configured to select a second main subject on abasis of only the tendency toward approach of the each subject.
 4. Thehand-held camera according to claim 3, wherein the subject distancechange determination circuitry is configured to determine that the eachsubject that is approaching on average, cumulatively, or continuously isa subject with the tendency toward approach, as a detection result ofthe temporal change of the distance, and wherein the main subjectselection circuitry is configured to select a part or all of the subjectthat is determined to have the tendency toward approach as the secondmain subject.
 5. The hand-held camera according to claim 3, wherein thesubject distance change determination circuitry is configured to detectthe temporal change of the distance of the each subject in each divisionregion in the image.
 6. The hand-held camera according to claim 5,wherein the subject distance change determination circuitry isconfigured to determine a division region at which the each subject islocated in a second image before a unit time, and detect the temporalchange of the distance of the each subject as a difference between afirst subject distance of the division region that has been determinedand a second subject distance of a division region of the image that isa current processing target image.
 7. The hand-held camera according toclaim 5, wherein the subject distance change determination circuitry isconfigured to determine that the each subject of the division regionthat is approaching on average, cumulatively, or continuously is asubject with the tendency toward approach, as a detection result of thetemporal change of the distance, and wherein the main subject selectioncircuitry is further configured to select the second main subject on abasis of information regarding the division region in which the subjectis determined to have the tendency toward approach.
 8. The hand-heldcamera according to claim 7, wherein the main subject selectioncircuitry is configured to group a different division region in which asame subject as the subject contained in the division region that isdetermined to have the tendency toward approach, and set an image rangeserving as the second main subject based on a range of one or moredivision regions that have been grouped.
 9. The hand-held cameraaccording to claim 8, wherein the main subject selection circuitry isconfigured to group the division region and the different divisionregion for which a distance difference of a subject distance is within apredetermined value and which is a region continuous with the divisionregion.
 10. The hand-held camera according to claim 1, wherein, todetect the temporal change of the distance, the subject distance changedetermination circuitry is further configured to detect the temporalchange of a size of a subject image in the image.
 11. The hand-heldcamera according to claim 1, wherein, to detect the temporal change ofthe distance, the subject distance change determination circuitry isfurther configured to detect the temporal change of a size of a subjectimage in the image, the subject image being one or more of candidateimages extracted in the image.
 12. The hand-held camera according toclaim 1, further comprising: an attribute identification circuitryconfigured to identify an attribute of the subject, and output attributeinformation on a basis of the attribute that has been identified,wherein the subject distance change determination circuitry isconfigured to change a determination condition of the tendency towardrecession of the subject according to the attribute information.
 13. Thehand-held camera according to claim 1, wherein the control circuitry isfurther configured to control a memory to store a still image includingthe first main subject responsive to receiving an image capture input.14. A method for controlling a hand-held camera, the method comprising:capturing, with an image sensor, an image; detecting, with a subjectdistance change determination circuitry, a temporal change of a distancefrom an imaging position to each subject of a plurality of subjectspresent in the image; determining, with the subject distance changedetermination circuitry, a tendency toward recession of the each subjectof the plurality of subjects with respect to the imaging position on abasis of the temporal change of the distance from the imaging positionto the each subject; selecting, with a main subject selection circuitry,a first main subject on a basis of the tendency toward recession of theeach subject of the plurality of subjects that has been determined bythe subject distance change determination circuitry, wherein the firstmain subject has a tendency towards recession; and tracking, with acontrol circuitry, the first main subject in response to the mainsubject selection circuitry selecting the first main subject.
 15. Themethod according to claim 14, further comprising: determining, with thecontrol circuitry, whether the first main subject is being tracked whenthe image is captured by the image sensor, and controlling, with thecontrol circuitry, a memory to store a portion of the image that iscaptured as a still image in response to determining that the first mainsubject is being tracked when the image is captured.
 16. The methodaccording to claim 14, wherein the subject distance change determinationcircuitry is further configured to determine a tendency toward approachof the each subject to the imaging position on a basis of informationregarding the temporal change of the distance of the each subject, andwherein the main subject selection circuitry is further configured toselect a second main subject on a basis of only the tendency towardapproach of the each subject.
 17. The method according to claim 16,wherein the subject distance change determination circuitry isconfigured to determine that the each subject that is approaching onaverage, cumulatively, or continuously is a subject with the tendencytoward approach, as a detection result of the temporal change of thedistance, and wherein the main subject selection circuitry is configuredto select a part or all of the subject that is determined to have thetendency toward approach as the second main subject.
 18. The methodaccording to claim 16, wherein the subject distance change determinationcircuitry is configured to detect the temporal change of the distance ofthe each subject in each division region in the image.
 19. The methodaccording to claim 18, wherein the subject distance change determinationcircuitry is configured to determine a division region at which the eachsubject is located in a second image before a unit time, and detect thetemporal change of the distance of the each subject as a differencebetween a first subject distance of the division region that has beendetermined and a second subject distance of a division region of theimage that is a current processing target image.
 20. A non-transitorycomputer-readable medium comprising instructions that, when executed byan electronic processor, cause the electronic processor to perform a setof operations comprising: capturing an image; detecting a temporalchange of a distance from an imaging position to each subject of aplurality of subjects present in the image; determining a tendencytoward recession of the each subject of the plurality of subjects withrespect to the imaging position on a basis of the temporal change of thedistance from the imaging position to the each subject; selecting afirst main subject on a basis of the tendency toward recession of theeach subject of the plurality of subjects that has been determined,wherein the first main subject has a tendency towards recession; andtracking the first main subject in response to selecting the first mainsubject.